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Abstract 


The  goal  of  machine  vision  is  to  allow  intelligent  systems  to  describe  the  world  around  them  b>  the  inierpre' 
lation  of  images.  The  difficulty  is  that  vision  is  a  very  complex  process,  smce  images  may  contain  shadows, 
highlights,  interreflections,  and  other  phenomena.  Images  are  created  through  the  interaction  of  light  with 
the  world;  therefore,  any  vision  system  that  is  to  understand  images  must  have  a  model  of  those  interactions. 
By  using  physics-based  models  to  describe  image  formation,  we  can  analyze  images  in  a  systematic  way. 

In  applying  physical  models  to  machine  vision,  one  of  the  key  tools  has  been  color  histogram  analysis.  A 
color  histogram  shows  the  variation  of  colors  observed  within  the  scene.  In  the  mid-1980s,  it  was  recognized 
that  the  color  variation  for  a  single  inhomogeneous  surface  can  be  modeled  as  a  regular  physical  process 
with  a  planar  distribution  in  color  space.  The  identification  of  this  plane  and  the  vectors  that  define  it  leads 
directly  to  an  analysis  of  object  color  and  illumination  color.  However  there  is  much  more  to  be  said  about 
color  histograms.  The  colors  do  not  fall  randomly  in  a  plane,  but  form  clusters  at  specific  pomts  in  color 
space.  The  colors  in  the  histogram  relate  not  only  to  the  color  of  the  object  and  the  illumination,  but  also  to 
non-color  properties  of  surface  roughness  and  imaging  geometry. 

The  color  histogram  for  an  object  illunainatedby  a  single  source  may  be  broken  into  two  clusters  of  pixels,  the 
body  reflection  cluster  and  the  highlight  cluster.  The  highlight  cluster  consists  of  those  pixels  that  correspond 
to  the  highlight  area  of  the  object,  whereas  the  body  reflection  cluster  consists  of  those  pixels  that  do  not 
show  an  appreciable  amount  of  highlight.  The  length  of  the  highlight  cluster  is  strongly  dependent  upon  the 
roughness  of  the  object,  since  smoother  objects  have  brighter  highlights.  The  width  of  the  highlight  cluster 
is  also  strongly  dependent  upon  roughness,  because  highlights  on  rough  surfaces  cover  a  greater  range  of 
reflectance  angles.  The  position  where  the  highlight  and  body  clusters  intersect  shows  the  magnitude  of  the 
body  reflection  component  in  the  highlight  pixels,  which  depends  upon  the  imaging  geometry.  However,  the 
intersection  is  also  affected  to  some  degree  by  the  object’s  roughness,  whereas  the  length  and  width  mea¬ 
surements  are  in  turn  affected  by  imaging  geometry.  An  understanding  of  the  interplay  of  these  factors  also 
shows  how  they  affect  estimates  of  the  illumination  color.  The  result  is  a  set  of  interdependent  relationships 
between  scene  parameters  and  histogram  features. 

Furthermore,  analysis  of  color  histograms  may  be  extended  to  images  that  contain  interreflection.  Applying 
the  dichromatic  reflection  model  to  interreflection  betw'een  two  objects  shows  how  color,  imaging  geometry’, 
and  surface  roughness  affect  the  color  histogram.  By  measuring  the  dimensions  of  the  resulting  clusters, 
additional  information  about  scene  characteristics  may  be  obtained  that  is  not  available  in  simpler  scenes. 

This  thesis  describes  an  algorithm  for  analyzing  color  histograms  that  yields  estimates  of  the  properties  of 
object  roughness,  object  albedo,  and  object  color,  as  well  as  illumination  position,  illumination  intensity' 
and  illumination  color.  The  relationship  between  these  scene  parameters  and  the  histogram  measurements 
is  complex  and  caimot  be  solved  analytically.  Instead,  the  solution  is  obtained  by  interpolating  between 
histogram  measurements  from  images  with  known  scene  parameters.  The  algorithm  is  tested  on  simulated 
images,  and  the  values  recovered  fay  the  program  compare  very  favorably  with  the  actual  parameters  used 
to  simulate  the  scene.  The  algorithm  is  also  tested  on  real  images,  and  the  calculated  values  are  reasonably 
close  to  estimates  of  the  scene  parameters  obtained  by  other  means. 

The  algorithm  presented  here  works  very  quickly,  and  requires  only  a  single  color  image.  This  method  may 
be  applied  to  such  varied  tasks  as  surface  inspection  and  object  recognition.  It  demonstrates  the  importance 
of  considering  laws  of  reflection  in  explaining  color  variation,  and  shows  why  color  is  such  a  valuable 
feature.  This  type  of  analysis  of  color  appearance  brings  us  closer  to  the  goal  of  allowing  machines  to 
interpret  images. 
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Chapter  1 


Introduction 


The  goal  of  machine  vision  is  to  allow  intelligent  systems  to  perceive  the  world  around  them.  Vision  is 
a  very  rich  sense  which  provides  much  information,  yet  it  is  also  complex  due  to  shadows,  highlights, 
interreflections,  and  other  phenomena.  In  order  to  understand  how  light  created  a  particular  image,  a  vision 
system  will  need  a  model  of  the  way  light  interacts  with  the  world.  Therefore,  a  continuing  area  of  research 
in  machine  vision  has  been  the  development  of  ever  more  sophisticated  models  of  image  formation,  that  are 
complex  enough  to  capnne  the  richness  of  optical  phenomena  seen  in  real  images,  and  yet  simple  enough 
that  they  can  be  inverted  to  recover  information  about  scenes.  Since  image  formation  involves  the  projection 
of  a  three  dimensional  scene  into  two  dimensional  image  data,  the  problem  is  of  course  under-constrained. 
Yet  the  human  visual  system  is  able  to  recover  an  astonishing  amount  of  scene  information  from  images.  So 
far,  machine  vision  is  not  nearly  so  sophisticated,  but  researchers  have  made  a  lot  of  progress  in  showing  how 
images  may  be  automatically  interpreted  to  recover  useful  information  about  the  scene.  Color  images  have 
proved  especially  useful  for  analyzing  images  for  such  tasks  as  object  classification  and  scene  segmentation. 

This  thesis  will  show  how  color  information  from  an  image  can  be  used  to  recover  interesting  properties 
of  the  scene.  These  properties  include  the  color  of  objects  and  the  color  of  the  illumination,  but  also  non¬ 
color  properties  such  as  surface  roughness  and  illumination  position.  This  is  done  by  exploiting  the  structure 
inherent  in  color  histograms.  This  thesis  will  show  that  the  dimensions  of  the  color  histogram  are  related 
in  a  precise,  quantitative  way  to  scene  properties;  and  that  measurements  of  histogram  dimensions  may  be 
used  to  recover  estimates  of  scene  parameters.  The  result  is  a  program  that  automatically  calculates  these 
scene  parameters  using  only  a  histogram  from  a  single  color  image. 
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CHAPTER]  ISTRODUCTION 


1.1  The  Use  of  Color  Histograms  in  Image  Understanding 


Color  histograms  have  long  been  used  by  the  machme  vision  communm-  m  image  understanding.  Color  has 
been  thought  of  as  an  important  property  of  objects,  since  it  can  be  used  for  segmentation  and  classification 
Unfommately  color  is  not  uniform  for  all  objects  of  a  similar  class,  nor  even  across  a  single  object.  TTiat 
would  make  the  segmentation  and  classification  problem  too  easy!  Color  variation  has  come  to  be  expected 
in  images,  and  thus  vision  researchers  have  been  working  on  modeling  this  variation. 

The  color  histogram  provides  a  method  for  representmg  color  vanation  in  a  straight-forward  way.  A 
histogram  is  created  from  the  color  values  at  each  pixel;  it  shows  for  each  point  in  color  space  how  many 
pixels  exhibit  that  color.  In  many  images,  the  colors  tend  to  form  clusters  in  the  histogram,  one  for  each 
object  in  the  image.  The  earliest  uses  of  color  histograms  modeled  the  histogram  as  a  Gaussian  cluster  in 
color  space  [Haralick  and  Kelly,  1969].  By  manual  or  automatic  analysis  of  the  histogram,  the  shape  of 
each  cluster  is  found.  Then  each  pixel  in  the  image  is  assigned  to  the  cluster  that  is  closest  to  the  pixel  color 
in  color  space.  For  example,  in  a  road  following  task,  pixels  that  correspond  to  grass  could  be  modeled  as 
having  a  canonical  color  of  green  with  some  possible  deviation  from  this  color.  The  color  variation  is  then 
modeled  as  a  probability  distribution,  so  that  the  further  from  the  characteristic  color,  the  less  likely  that  it 
is  a  grass  pixel.  The  problem  then  is  to  choose  the  size  and  shape  of  the  distribution.  If  the  size  is  too  large, 
many  non-grass  pixels  will  be  incorrectly  included:  if  it  is  too  small,  a  large  number  of  grass  pixels  will  be 
wrongly  rejected.  Figure  1.1  shows  a  diagram  of  this  idea.  This  type  of  classification  of  regions  by  color 
histogram  analysis  is  being  used  today  in  robotics  applications  such  as  autonomous  navigation  [Crisman 
and  Thorpe,  1991].  However,  this  approach  assumes  a  random  variation  in  color,  and  so  would  not  be  well 
suited  to  systematic  variations  in  colors  such  as  would  be  associated  with  shading  or  highlights. 

A  more  pow^ful  approach  for  handling  systematic  variations  in  color  is  to  consider  the  physics  of 
light  reflection  from  objects.  In  1984  Shafer  showed  that  for  dielectric  materials  with  highlights,  the  color 
histogram  associated  with  a  single  object  forms  a  plane  [Shafer,  1984].  This  plane  is  defined  by  two  color 
vectors:  a  body  reflection  vector  and  a  surface  reflection  vector.  In  a  diagram  accompanying  this  idea 
(figure  1 .2),  he  visualized  that  the  histogram  would  fill  out  a  parallelogram.  The  paper  did  not  describe  how 
to  determine  these  two  vectors.  In  fact,  for  any  planar  histogram,  there  are  many  sets  of  two  vectors  that 
could  define  it. 


However,  in  1987  Klinker  and  Gershon  independently  observed  that  the  color  histogram  does  not 
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Figure  1.1:  Gaussian  histogram  as  described  by  Haralick  and  Kelly.  1969 
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Figure  1.2:  Planar  histogram  as  described  by  Shafer,  1984 
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Figrire  1.3:  T-shaped  histogram  as  described  by  Klioker.  1987 

uniformly  fill  the  plane,  but  instead  forms  a  T-shape  or  dog-leg  in  color  space  [Klinker  ef  al..  1 987].[Gershon. 
1987],  Figure  1.3  shows  that  this  color  histogram  is  composed  of  two  linear  clusters,  one  correspondmg 
to  pixels  that  exhibit  mostly  body  reflection  and  one  corresponding  to  pixels  that  exhibit  mostly  surface 
reflection.  This  T-shape  made  it  possible  to  identify  characteristic  body  reflection  and  illumination  colors. 

In  1988,  Healey  showed  that  the  number  of  dimensions  occupied  by  the  histogram  may  be  used  to 
determine  material  type  [Healey,  1988].  Metals  do  not  exhibit  any  body  reflection;  they  show  only  surface 
reflection.  Thus  the  color  histogram  of  a  metal  object  will  have  only  a  one-dimensional  distribution  in  color 
space,  in  contrast  to  the  two  dimensional  distribution  for  dielectrics  with  highlights.  Examining  the  color 
histogram  allows  the  automatic  classification  into  metal  and  non-metal. 

The  shape  of  color  variation  has  also  been  used  for  recognition  tasks.  In  1990  Swain  used  color 
histograms  for  indexing  into  a  large  database  of  multi-colored  objects  [Swain,  1990].  The  distribution  of 
colors  on  each  object  is  invariant  with  rotations  and  translations  in  the  image  plane,  and  changes  slowly  with 
changes  in  view  angle  and  scale,  allowing  the  histogram  in  an  image  to  be  matched  to  the  appropriate  model 
in  the  database.  Funt  and  Finalyson  have  proposed  histogramming  color  ratios  instead  of  color  values  to 
allow  for  changes  m  incident  illumination  color  [Funt  et  al.,  1991].  Although  these  histogram  techniques 
allows  real-time  identification  and  location  of  complex  patterned  objects  as  cereal  boxes  and  striped  clothing, 
they  do  not  make  any  other  analysis  of  scene  characteristics. 

There  has  also  been  some  work  in  analyzing  histogram  shape  when  inteirefleclion  is  present.  Fimt  and 
Drew  have  observed  that  for  intcrreflection  between  diffuse  surfaces,  the  pixels  for  each  surface  will  lie  on 
a  plane  defined  by  the  color  vector  of  the  object’s  color  and  the  color  vector  defined  by  the  inteneflection 
color  [Funt  and  Drew,  1991].  Under  some  circumstances,  the  two  objects  will  have  different  planes.  If 
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the  intersection  of  the  two  planes  can  be  found,  the  interreflection  can  automatically  be  removed  from  the 
pictures.  The  method  does  not  require  shape  infonaation  about  the  surfaces,  other  than  the  assumption 
that  they  are  convex.  Since  the  objects  are  assumed  to  be  diffuse  surfaces,  the  method  does  not  consider 
highlights  or  interrefiection  involving  highlights. 

Bajcsy  and  Lee  have  described  an  algorithm  that  segments  images  that  contain  both  highlights  and 
interreflection  [Bajcsy  et  al.,  1989],  The  analysis  is  done  on  two-dimensional  histograms  in  a  color  space 
similar  to  chromaticity.  First  color  constancy  is  performed  using  a  white  reference  object.  After  this  step, 
a  highlight  on  an  object  w'ill  have  the  same  hue  as  the  body  color,  so  when  the  image  is  segmented  by 
hue,  highlight  and  body  regions  will  be  kept  together.  Since  the  highlight  is  a  whitened  region  (after  color 
constancy!  it  will  have  a  different  saturation  value  than  the  body  region,  so  any  highlighted  areas  may 
be  identified  by  a  further  segmentation  by  saturation.  Interrefiection  is  recogmzed  when  it  causes  small 
deviations  from  the  object’s  hue.  The  algorithm  has  been  applied  to  actual  images;  however  in  these  scenes 
the  hue  did  not  vary  significantly  in  interrefiection  areas.  Other  types  of  images  can  show  dramatic  hue  shifts 
due  to  interrefiection. 

Color  histograms  have  been  used  for  many  years  in  image  understanding  research,  and  have  proved  very- 
useful  in  a  variety  of  tasks.  Several  researchers  have  used  physics-based  models  to  explain  the  relationship 
between  scene  characteristics  and  histogram  shape.  However,  until  now.  physics-based  models  have  not 
been  fully  exploited  to  explain  the  detailed  structure  of  the  histogram. 


1.2  Physics-Based  Models  for  Image  Understanding 


Physics-based  models  are  appealing  since  they  describe  a  systematic  relationship  between  scene  parameters 
and  image  values.  This  approach  makes  it  possible  to  analyze  image  features,  such  as  highlights,  which 
cannot  be  modeled  well  with  statistics.  Physics-based  vision  has  become  increasingly  popular  in  the  last 
few  years  and  has  been  applied  to  a  variety  of  tasks. 

A  physics-based  model  used  extensively  in  color  vision  research  is  the  "dichromatic  refieciion  model", 
first  proposed  in  [Shafer,  1984],  The  model  states  that  the  light  reflected  from  a  dielectric  object  is  the 
sum  of  two  components:  a  surface  component  and  a  body  component.  The  model  further  states  that  each 
component  in  turn  may  be  separated  into  a  color  term  that  depends  only  on  wavelength,  and  a  geometric 
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term  that  depends  only  upon  viewing  geomeirj’.  This  equation  gives  nse  to  the  planar  distribution  predicted 
by  Shafer  [Shafer.  1984]  and  observed  by  both  Klinker  [Klinker^r  al.,  1987]  and  Gershon  [Gershon,  1987], 

The  spectral  component  of  the  surface  reflection  is  often  modeled  as  being  the  same  as  that  of  the 
incident  light.  This  assumption  is  sometimes  called  the  Neutral  Interface  Reflectance  (NIR)  model  [Lee, 
1988].  The  spectral  component  of  the  body  reflection  is  modeled  as  being  the  product  of  the  object  color 
and  the  illumination  color.  This  means  that  if  the  illumination  color  can  be  identified,  then  its  influence  may 
be  divided  out  of  the  body  color  to  give  the  object’s  reflectance.  This  provides  a  method  for  color  constancy 
imderchangmg  illumination  conditions  [Klinker.  1988],  (D’Zmura  and  Lennie,  1986],  [Healey  and  Bmford. 
1987a],  [Toininaga  and  Wandell,  1989]. 

Recently  Sato  and  Ikeuchi  have  analyzed  image  sequences  of  shiny  surfaces  taken  while  the  light  source 
is  moving  [Sato  and  Ikeuchi,  1992].  Each  pixel  is  individually  separated  into  body  reflection  and  surface 
reflection  components  using  singular  value  decomposition,  and  the  surface  normal  at  that  point  is  recovered. 
The  method  analyzes  the  rank  of  the  matrix  formed  from  the  sequence  of  color  values;  since  each  pixel  from 
a  dielectric  is  composed  of  two  reflection  components,  the  rank  will  be  two.  The  method  also  works  on 
metal  objects,  when  the  rank  of  the  matrix  is  one  (since  metals  only  exhibi:  surface  reflection).  However  the 
method  requires  a  sequence  of  images  with  lights  at  different,  known  positions. 

The  roughness  of  an  object  also  has  an  important  effect  on  its  appearance.  When  something  appears  shiny. 
It  is  usually  assumed  that  the  suitace  is  optically  very  smooth.  The  amount  of  shininess  is  usually  associated 
with  the  brightness  and  sharpness  of  highlights.  Although  several  people  have  worked  on  analyzing  surface 
roughness  from  images,  no  work  has  previously  been  done  on  how  roughness  affects  the  appearance  of  color 
histograms.  This  is  not  surprising  since  the  previous  work  in  roughness  has  mainly  focussed  on  black  and 
white  images. 

In  order  to  analyze  surface  roughness  from  images,  a  panicular  model  of  surface  roughness  and  its  affect 
on  appearance  is  needed.  In  [Mundy  and  Porter,  1980]  and  [Porter  and  Mundy,  1981]  the  roughness  of  metal 
surfaces  is  estimated  using  the  Beckmann-Spizzichino  model[Beckmann  and  Spizzichino.  1963].  The  work 
of  [Healey  and  Binford,  1987b]  used  the  Torrance-Sparrow  model  to  analyze  highlights  and  determine  local 
shape  from  highlights  [Torrance  and  Sparrow.  1967].  However,  this  method  did  not  provide  an  estimate  of 
surface  roughness. 

In  [Nayar  et  al.,  1991],  the  Beckmaim-Spizzichino  model  is  compared  with  the  Torrance-Sparrow;  it  i.s 
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shown  that  they  predict  essentially  the  same  surface  reflection  when  surfaces  are  reasonably  rough.  This 
work  also  proposes  a  hybrid  model  that  incorporates  both  these  roughness  models.  In  [Dceuchi  and  Sato, 
1990]  and  [Solomon  and  Ikeuchi,  1992]  roughness  parameters  are  estimated  using  the  Torrance-Spanow 
model.  Shape  information  is  provided  by  a  range  finder  [Ikeuchi  and  Sato,  1990]  or  by  4-light  photomemc 
stereo  [Solomon  and  Ikeuchi,  1992],  and  the  model  parameters  are  determined  using  the  least  squares  fit  to 
the  image  brightness  and  shape  data.  These  techniques  all  consider  object  shape  in  recovermg  roughness 
estimates. 

Interreflection  presents  a  very  difficult  problem  for  machme  vision,  and  it  is  only  recently  that  it  has  been 
smdied  in  actual  images.  However,  it  too  can  be  explained  with  physical  models.  The  theoretical  nature 
of  the  problem  was  examined  a  considerable  time  ago  by  Horn.  In  [Horn,  1977]  a  denvation  is  given  for 
the  interreflection  of  two  semi-infinite  planes  that  intersect  at  right  angles.  These  requirements  are  relaxed 
to  allow  the  planes  to  meet  at  angles  other  than  90  degrees  and  to  allow  the  planes  to  have  a  finite  length 
from  the  intersection  (although  still  infinite  in  the  direction  along  the  intersection).  The  calculation  is  said 
to  closely  approximate  the  actual  behavior  close  to  a  comer  for  real  intersections  of  flat  objects.  However 
the  usefulness  of  this  derivation  is  limited  by  the  assumption  t  it  the  surfaces  are  Lambertian. 

Another  theoretical  application  of  physical  models  to  interreflection  is  a  variation  of  the  dichromatic 
reflection  model  [Brill,  1989].  The  idea  is  that  where  the  number  of  sensors  is  one  greater  than  the  number 
of  different  types  of  reflection  functions,  the  range  of  colors  seen  on  each  object  will  span  a  (hyper)plane  in 
sensor  space.  Therefore  the  normal  to  this  plane  may  be  used  to  segment  a  scene.  Although  the  main  focus 
of  the  paper  is  upon  matte  objects  with  two  light  sources,  it  mentions  that  the  theory  is  equally  applicable  to 
the  situation  where  one  light  source  illuminates  a  dielectric  object  with  a  highlight  (the  dichromatic  reflection 
model),  and  to  the  situation  where  a  matte  object  is  illuminated  by  a  single  light  source  plus  interreflection 
from  a  nearby  (matte)  object.  This  work  is  theoretical  in  nature  and  has  not  been  applied  to  any  actual 
images  containing  interreflection. 

The  observation  has  been  made  that  outdoors  where  the  sun  is  yellow  and  the  sky  is  blue,  shadows  tend 
to  look  bluish  [Sloan,  1977].  Although  this  is  a  very  qualitative  statement,  it  implicitly  contains  the  idea  that 
shadow  regions  are  lit  by  interreflection  from  the  atmosphere  and  that  this  effect  is  more  important  m  areas 
not  lit  by  direct  illumination  from  the  sun. 

The  idea  that  interreflection  plays  an  imponant  part  in  the  appearance  of  shadow  regions  is  also  given 
in  [Forsyth  and  Zisserman,  1989],  In  this  paper,  an  equation  for  interreflection  is  derived  and  the  simulated 
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results  are  compared  with  the  results  of  actual  images.  The  authors  also  show  that  inierreflection  is  more 
pronounced  for  objects  with  high  albedo  (white)  than  for  objects  with  low  albedo  (black).  However  this  work 
looked  at  Lambertian  reflection  only.  Shiny  black  objects  will  usually  show'  considerable  inierreflection  ot 
other  objects  in  the  scene. 

In  19d0,  Nayar  demonstrated  the  usefulness  of  physical  models  for  recovering  scene  information  from 
real  images  with  interreflection  [Nayar  et  al.,  1990].  This  paper  presents  an  algorithm  that  determines  shape 
and  albedo  of  objects,  and  that  explicitly  predicts  and  accounts  for  mterrefiection.  An  initial  estimate  of 
surface  shape  and  reflectance  is  made  w'ithout  accounting  for  interreflection,  That  initial  estimate  is  used  to 
calculate  the  amount  of  expected  interreflection  by  using  the  radiosity  methodiGoral  et  al..  1984].  This  in 
turn  allows  a  better  estimate  of  shape  and  reflectance.  The  process  is  repeated  until  the  solution  converges. 
The  algorithm  uses  black  and  w’hite  pictures  rather  than  color,  and  uses  multiple  exposures  with  the  light 
source  in  different  positions  to  obtain  shape  information.  It  also  relies  upon  the  objects  bemg  matte,  without 
any  highlights  to  cause  interreflection. 

As  mentioned  in  section  1.1,  Funt  and  Drew  have  considered  interreflection  between  matte  surfaces. 
Their  work  uses  a  “one-bounce”  model,  considering  light  that  is  interreflected  at  most  once  between  surfaces . 
In  their  terminology,  light  that  comes  from  the  illumination  and  is  reflected  by  an  object  directly  into  the 
camera  is  called  “no-bounce”.  They  analyzed  interreflection  using  color-space  analysis  of  histogram  [Funt 
and  Drew’,  1991]  and  by  using  the  calculus  of  variations  [Drew  and  Funt,  1992],  Jang  also  analyzed 
interreflection  between  matte  objects,  using  a  model  that  considers  at  most  once  botmce  of  interreflection 
[Jang,  1991].  Both  this  method  and  those  of  Funt  and  Drew  rely  upon  having  a  pre-segmented  image  and 
identification  of  the  colors  involved  m  interreflection  ahead  of  time. 

Only  the  work  of  Lee  considers  interreflection  among  shiny  objects  that  exhibit  highlights  [Lee,  1991]. 
The  model  used  is  an  extension  of  the  dichromatic  reflection  model,  that  considers  the  four  types  of 
interreflection  that  are  inherent  in  a  one-boimce  model  between  shiny  surfaces.  Although  all  four  types  are 
listed,  their  relative  strengths  are  not  addressed.  It  also  assumes  that  the  hue  change  caused  by  interreflection 
is  very  small,  which  is  not  true  in  many  images.  This  work  did  not  consider  the  effect  of  roughness  on 
interreflection  and  provided  no  quantitative  analysis  of  interreflection. 

Physics-based  models  have  proved  very  useful  in  machine  vision  research,  for  tasks  ranging  from 
analyzing  color  to  estimating  roughness.  However,  physics-based  models  may  be  used  in  a  more  complete 
way  to  make  more  detailed  predictions  about  color  appearance,  and  to  recover  more  information  about  the 
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1.3  A  More  Complete  Analysis  of  Color  Histograms 


One  problem  with  previous  work  in  analyzing  color  has  been  that  it  did  not  fully  consider  the  effect  of 
non-color  parameters  on  color  appearance.  It  is  immediately  obvious  that  the  color  of  an  object  has  an  effect 
upon  the  color  observed  in  an  image  of  that  object.  A  red  object  is  expected  to  appear  red,  a  yellow  object 
is  expected  to  appear  yellow,  and  so  forth.  The  study  of  color  constancy  in  machine  vision  has  emphasized 
the  fact  that  the  color  of  illumination  also  has  an  effect  on  color  appearance.  People  outside  the  color  and 
vision  communities  are  often  unaware  of  this,  since  human  color  vision  is  advanced  enough  to  compensate 
for  almost  all  changes  in  illumination.  Nevertheless,  the  impact  of  illumination  color  on  image  appearance  is 
obvious  to  those  who  work  with  color  images,  and  is  a  logical  consequence  of  the  physics  of  color  reflection. 

This  thesis  addresses  the  way  other  scene  parameters  affect  the  color  distribution  across  objects.  Since 
optical  roughness  determines  whether  a  surface  looks  matte  or  shiny,  it  is  obvious  that  it  affects  scene 
appearance  in  some  way.  However  previous  researchers  that  have  looked  at  roughness  have  only  considered 
its  effect  on  image  intensity.  By  considering  the  combination  of  both  body  and  surface  reflection  components 
in  color  images,  this  thesis  shows  that  smooth  surfaces  exhibit  colors  that  are  not  seen  anywhere  on  rough 
surfaces  and  vice  versa. 

Similarly,  it  has  long  been  known  that  highlights  move  when  imaging  geometry  changes.  This  has  posed 
substantial  problems  for  stereo  algorithms  since  the  two  cameras  are  viewing  objects  in  the  scene  from 
somewhat  different  angles.  This  makes  it  difficult  to  find  correspondences  between  the  two  images  since 
highlights  are  not  a  stable  feature.  This  thesis  will  show  that  a  change  in  imaging  geometry  not  only  changes 
the  position  of  highlights,  but  their  actual  colors  as  well.  The  colors  observed  in  a  highlight  viewed  from 
straight  on  are  actually  not  present  when  the  same  highlight  is  seen  from  an  oblique  angle  and  vice  versa. 

Since  the  observed  colors  are  affected  by  both  surface  roughness  and  imaging  geometry,  the  obvious 
hope  is  that  the  color  can  be  analyzed  to  recover  those  scene  parameters.  Physics-based  reflection  models 
show  that  the  color  varies  in  a  systematic  way  as  roughness  and  imaging  geometry  are  varied.  This  thesis 
will  show  how  this  systematic  variation  can  be  exploited  to  recover  these  scene  parameters. 


An  understanding  of  the  effects  of  roughness  and  imaging  geometry  upon  color  distribution  also  show 
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how  to  estimate  the  illumination  color  from  the  image  of  a  single  object  with  highlights.  Although  it  has  long 
been  recognized  that  the  pixels  from  this  object  will  lie  in  a  plane  defined  by  the  body  reflection  and  surface 
reflection  colors,  the  problem  has  been  in  identifying  these  two  vectors  from  the  infiniie  set  of  vectors  that 
lie  in  the  dichromatic  plane. 

This  problem  can  be  solved  if  there  is  more  than  one  object  with  highlights  in  the  scene;  then  the 
surface  reflection  vector  can  be  identified  by  the  intersection  of  the  dichromatic  planes.  If  there  is  only 
one  object,  a  vector  may  be  obtained  by  the  best  fit  to  the  highlight  cluster  in  the  color  histogram.  This 
vector  was  used  by  KlinJcer  to  approximate  the  surface  reflection  cluster  [Klinker,  1988j.  The  degree  to 
which  this  approximation  is  accurate  depends  upon  the  surface  roughness  and  imaging  geometry’.  Varying 
these  parameters  has  a  systematic  effect  upon  the  cluster  appearance,  which  may  be  analyzed  to  recover  an 
unproved  estimate  of  the  iilummation  color. 

Research  in  color  images  has  traditionally  only  been  able  to  recover  a  relative  color  measurement  for 
objects  and  illumination.  Estimates  of  object  color  considered  only  chromaticity,  not  absolute  reflectance, 
since  it  was  considered  impossible  to  tell  a  bright  source  illuminating  a  dark  object  from  a  dim  source 
illuminating  a  light  object.  However  by  considering  the  appearance  of  highlights,  it  is  possible  to  show  that 
there  is  a  definite  difference  in  appearance  for  these  two  scenarios.  This  thesis  will  show  how  the  differeuC'* 
may  be  exploited  to  distinguish  between  object  albedo  and  illumination  brighmess. 

Whenever  there  is  more  than  one  object  in  the  scene,  there  is  the  possibility  for  interreflection  between 
the  two  objects.  The  usual  assumption  made  in  image  understanding  is  that  a  scene  is  illuminated  by  a  single 
light  source  which  emits  light  onto  the  scene.  This  light  is  reflected  by  individual  objects  in  the  scene,  and 
some  of  that  light  enters  the  camera.  In  this  model  the  energy  reflected  by  an  object  does  not  interact  with 
the  light  source  nor  with  other  objects  in  the  scene.  However,  even  simple  pictures  with  a  very  small  number 
of  objects  often  exhibit  interreflection. 

Most  of  the  work  on  the  problem  of  interreflection  has  only  considered  light  which  which  is  interreflected 
once  between  matte  objects.  This  thesis  will  refer  to  this  as  the  “secondary  reflection”  model,  since  it 
considers  light  that  is  reflected  from  two  surfaces.  (By  contrast  Ftmt  refers  to  this  as  “one  bounce”  since  it  is 
interreflected  only  once  [Fxmt  and  Drew,  1991].)  More  than  two  reflections  are  obviously  possible,  but  the 
effect  of  additional  reflections  upon  the  image  are  shown  here  to  be  small  for  inhomogeneous  dielectrics. 

The  matte  assumption  is  appealing  since  researchers  only  have  to  consider  two  color  components; 
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primary  body  reflection  that  comes  directly  from  the  object,  and  secondary  body  reflection  that  came  first 
from  the  body  reflection  of  a  nearby  object.  As  Funt  points  out,  these  two  components  mean  that  pixels  form 
a  plane,  somewhat  analogous  to  the  dichromatic  plane  [Funt  and  Drew,  1991  ]. 

However  many  objects  that  exhibit  interreflection  have  some  amount  of  shuniness.  This  shininess  can 
have  a  noticeable  effect  upon  the  observed  iuterreflection.  Therefore  this  thesis  extends  the  Dichromatic 
Reflection  Model  to  the  case  of  secondary  interreflection.  By  considering  the  body  and  surface  reflection 
components  separately,  secondary  reflection  adds  four  additional  color  components  to  the  two  that  come  from 
primary  reflection.  The  relative  strengths  of  these  components  determine  the  degree  to  which  interreflection 
is  noticeable  in  images. 

Once  highlights  are  considered  in  interrefiection,  it  becomes  necessary  to  consider  the  amount  of 
shininess  in  order  estimate  the  relative  strengths  of  the  interrefiection  components.  Therefore  the  surface 
roughness  is  an  important  parameter  in  analyzing  interrefiection.  The  “hue  shifts”  that  are  observed  in 
interrefiection  will  depend  upon  objects’  surface  smoothness  as  well  as  their  colors.  This  thesis  will  show 
how  considering  the  components  of  interrefiection  due  to  surface  reflection  leads  to  a  better  understanding  of 
object  appearance,  and  may  give  some  additional  information  that  caimot  be  recovered  from  simpler  scenes 
without  interrefiection. 


1.4  Thesis  Outline 

This  thesis  shows  how  the  structure  of  color  histograms  may  be  exploited  to  recover  valuable  scene  infor¬ 
mation,  It  begins  in  chapter  2  with  an  explanation  of  the  reflection  models  used  in  this  work.  Chapter  3 
desQibes  how  the  histogram  for  a  single  object  is  related  to  various  scene  parameters.  Chapter  4  extends  this 
to  show  how  the  histogram  is  affected  by  mterreflection  from  a  nearby  object.  These  two  chapters  show  how 
scene  parameters  affect  the  color  distribution  in  images.  This  type  of  description  is  very  valuable,  but  does 
not  necessarily  mean  that  it  can  be  applied  to  vision  algorithms.  In  chapter  5,  this  thesis  begins  describing 
how  the  color  distribution  can  be  analyzed  to  recover  scene  parameters  from  color  variation.  Chapter  5 
shows  how  the  color  histogram  may  be  analyzed  if  an  idealized  sensor  could  be  employed.  Chapter  6  extends 
the  analysis  for  a  more  realistic  camera  that  has  characteristics  similar  to  those  used  by  vision  researchers 
today.  Chapter  7  shows  results  in  estimating  scene  properties  when  the  algorithm  s  applied  to  real  images. 
Chapter  8  summarizes  the  work  of  this  thesis,  discusses  the  contributions  made,  and  points  out  directions  of 
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futxire  research. 


Chapter  2 


Physics-Based  Models 


Much  of  machine  vision  has  evolved  from  classical  signal  processing.  This  approach  treais  image  data  as 
an  arbitraiy  2-dimensional  function  lix,y).  When  the  image  data  is  .  .vcd  tins  way.  it  is  natural  to  to  take 
derivatives,  apply  Fourier  transforms,  or  use  statistical  models.  This  idea  has  had  a  lot  of  success  in  areas 
such  as  edge  detection. 

The  problem  with  this  approach  is  that  for  real  images  the  function  is  not  arbitrary,  but  depends  upon 
physical  laws.  The  goal  of  an  image  understanding  system  is  not  really  to  find  edges  in  the  image,  but  to 
find  edges  in  the  world.  A  vision  model  based  on  they  physics  of  bow  highlights  are  formed  will  have  a 
much  better  chance  mterpreting  them. 

This  chapter  explains  the  physics-based  models  used  in  this  thesis.  The  first  section  describes  a  reflection 
model  for  objects  that  exhibit  both  matte  lefiection  and  highlights  as  a  result  of  two  separate  reflection 
processes.  Then  the  following  sections  desenbe  reflection  models  for  each  of  these  two  types  of  reflection. 


2.1  Dichromatic  Reflection  Model 


A  physics-based  model  used  extensively  in  color  vision  research  is  the  "Dichromatic  Reflection  Model", 
first  proposed  in  [Shafer.  19841.  'fhe  model  states  that  the  light  L  reflected  from  a  dielectric  object  is  the 
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Incident  light 


sum  of  two  components;  a  surface  component  Ls  and  a  body  component  Lb- 

L{\,e,,9r,6p)  =  Lb{X.e.-9r,ep)-i-Ls{\,e,.er.ep) 

This  corresponds  to  the  two  different  ways  in  which  light  is  reflected  by  inhomogeneous  dielectrics.  These 
surfaces  are  typically  composed  colorant  particles  dispersed  throughout  a  clear  medium.  This  model  applies 
to  plastics,  painted  surfaces,  ceramics,  paper,  cloth,  etc.  The  model  does  not  apply  to  homogeneous  surfaces 
such  as  metals,  which  exhibit  only  surface  reflection.  When  light  passes  from  the  air  into  the  medium  of  an 
inhomogeneous  dielectric,  a  change  in  the  index  of  refraction  causes  some  of  the  light  to  be  immediately- 
reflected  back  into  the  air.  The  rest  of  the  light  is  refracted  into  the  surface  where  it  is  bounced  around 
between  colorant  particles.  These  particles  selectively  absorb  certain  w-avelengths  of  light,  giving  rise  to  the 
object’s  characteristic  color.  A  diagram  of  this  process  is  shown  in  figure  2.1. 

Each  of  the  two  components  is  a  function  of  the  wavelength  of  light  (A)  and  the  angles  of  incidence  (0,j, 
reflection  {dr)  and  phase  angle  (6p).  These  angles  are  pictured  in  the  diagram  in  figure  2.2.  6s  is  the  angle 
between  the  surface  normal  and  the  perfect  specular  direction  (the  bisector  of  the  incidence  and  reflectance 
directions). 

The  Dichromatic  Reflection  Model  further  states  that  each  component  in  turn  may  be  separated  into  a 
color  term  c  that  depends  only  on  A,  and  a  geometric  term  m  that  depends  only  upon  6,,  9r,  and  dp. 

L{X,6i,dr,dp)  =  mb(0hOr,dp)Cb{X)  +  msidi,dr,dp)Cs{X) 

This  equation  gives  rise  to  a  planar  distribution.  Each  pixel  may  be  viewed  as  the  sum  of  some  amount  of 
body  reflection  plus  some  amount  of  surface  reflection.  This  idea  is  illustrated  in  in  figure  2.3. 


For  the  purposes  of  this  thesis,  the  surface  color  term  c^fA)  and  body  color  term  A)  will  only  capture 
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Figure  2.2:  Angies  that  define  imaging  geometry 


Figure  2.3:  Dichromatic  reflection  components  in  a  single  pixel 

relative  reflectance,  which  includes  hue  and  saturation  but  not  brightness.  The  brighmess  will  be  captured 
in  the  geometry  terms  nib  and  These  functions  of  wavelength  can  be  converted  into  RGB  triples  by 
multiplying  the  sensor  responsivities  for  red,  green,  and  blue  and  integrating  over  the  visible  spectrum. 

m  S  Cfr(A)5/?(A)dA 
Cb=  mb  S  Cb(A)SG(A)dA 

ffib  I  Cb{>^)SB{N)dX 

J  CsiX)SB{X)dX 

cl=  ms  S  Cs(.X)SG{X)dX 

ms  S  Cs(X)SB{X)dX 
The  vector  quantities  cl  and  cl  are  then  normalized  to  have  length  one. 

The  dichromatic  reflection  model  shows  how  certain  types  of  objects  exhibit  two  types  of  reflection, 
and  that  these  two  types  of  reflection  will  typically  have  different  hues.  This  hue  difference  means  the 
histogram  v/ill  fail  in  a  plane  in  the  color  space.  Furthermore,  the  two  reflection  processes  also  have  different 
geometrical  distributions  associated  with  the  physical  reflection  models. 


CHAPTER!.  PHYSICS-BASED  MODELS 


32 

2.2  Body  Reflection  Model 

The  body  reflection  is  usually  assumed  to  be  Lambertian.  Under  this  assumption,  a  point  on  an  object 
appears  equally  bright  from  any  reflectance  angle.  The  only  dependence  upon  imaging  geometry  is  that 
the  reflectance  is  proponional  to  the  cosine  of  the  incidence  angle.  The  Lambenian  model  gives  the  bod\ 
reflection  nib  component  as 


mb(ei..6r.dp)  =  -rBb  cosid,)  (2.1) 

where  is  the  angle  of  illumination  incidence.  The  gain  of  the  camera  in  converting  photons  measured  by 
the  CCD  array  into  pixel  values  is  represented  by  y.  The  brightness  of  the  body  reflection  is  represented 
by  the  term  Bb-  This  factors  in  both  the  reflectance  of  the  object  (albedo)  and  the  intensity'  of  the  light.  If 
the  intensity  of  the  illumination  is  known,  then  these  factors  may  be  separated  and  the  albedo  alone  may  be 
recovered. 

Recent  work  by  Wolff  has  shown  thit  that  the  Lambertian  assumption  is  very  good  for  a  large  range  of 
incidence  and  reflectance  angles;  however,  there  is  a  considerable  error  at  extreme  angles  such  as  are  seen 
at  occluding  edges  [Wolff,  1992],  Points  that  are  viewed  from  an  oblique  angle  {6r  close  to  90'^)  are  much 
less  bright  than  would  be  predicted  from  the  Lambertian  model.  Although  the  Wolff  model  has  a  slightly 
different  dependence  upon  incidence  angle  than  the  Lambertian  model,  both  models  predict  that  the  intensity 
falls  off  monotonically  as  the  incidence  angle  is  increased. 


2.3  Surface  Reflection  Model 

Unlike  the  body  reflection,  the  surface  reflection  component  is  very  dependent  upon  the  reflectance  angle. 
This  is  easily  noticed  by  watching  a  highlight  move  as  the  view  moves.  The  surface  reflection  also  falls  off 
much  more  quickly  than  body  reflection.  It  has  its  maximum  approximately  at  the  perfect  specular  angle 
where  the  angle  of  incidence  equals  the  angle  of  reflectance. 

The  spectral  component  of  the  surface  reflection  is  modeled  in  this  thesis  as  being  the  same  as  that  of 
the  incident  light.  This  is  an  approximation,  since  the  Fresnel  equation  that  governs  surface  reflection  shows 
that  the  reflection  is  dependent  upon  the  wavelength  of  the  incident  light.  However  this  dependence  is  weak 
across  the  visible  spectrum.  The  assumption  that  surface  reflection  color  is  identical  to  illumination  color  is 
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smoother  surface 


Global  Surface  Normal 


Local  Surface  Normals 


rougher  surface 


Figrire  2.4;  The  facet  model  of  surface  reflection 


a  common  one.  It  follows  from  this  assumption  that  once  the  illumination  vector  is  identified,  us  influence 
may  be  divided  out  of  the  body  color  to  give  the  object's  reflectance.  This  provides  a  method  for  color 
constancy  under  changing  illumination  conditions  [Klinker.  1988].  [D’Zmura  and  Lennie.  1986].  [Healey 
and  Binford,  1987a],  [Tominaga  and  Wandell,  1989]. 


This  thesis  uses  the  Torrance-Sparrow  model  of  scattering  [Torrance  and  Sparrow,  1967],  This  model 
views  surfaces  as  a  collection  of  tiny  facets,  each  of  which  may  have  a  local  surface  nonnal  that  is  different 
from  the  global  surface  nonnal.  For  smoother  surfaces,  most  of  the  facets  have  an  orientation  very  close  to 
the  global  surface  nonnal.  As  a  surface  becomes  rougher,  it  has  a  greater  ntimber  of  facets  that  are  tilted 
with  respect  to  the  global  nonnal.  This  is  illustrated  in  figure  2.4.  The  distribution  of  facet  normals  is 
typically  modeled  as  Gaussian,  with  cr  describing  the  standard  deviation.  The  distribution  is  also  assumed 
to  be  isotropic,  with  rotational  symmetry  about  the  surface  normal,  giving  a  probability  distribution  p  of 

where  B  is  the  tilt  of  the  facet  with  respect  to  the  global  surface  normal. 


A  small  value  of  cr,  such  as  1“^,  means  that  the  standard  deviation  of  facet  angles  is  small,  and  that  the 
surface  is  very  smooth.  A  large  value  of  a,  such  as  10°.  means  a  rough  surface.  The  facets  are  assumed  to 
be  an  order  of  magnitude  larger  than  the  wavelength  of  visible  light,  but  this  optical  roughness  is  still  much 
finer  than  the  roughness  that  could  be  felt  with  fingertips  or  that  would  be  used  for  texture  analysis.  For  the 
remainder  of  this  thesis,  it  will  be  assumed  that  the  facet  size  is  a  constant  for  the  surfaces  to  be  analyzed. 


The  Torrance-Sparrow  used  for  scattering  gives  the  amoimt  of  surface  reflection  as 


ms(.Bi,dr,dp) 


jg  EGi6i,dr,6p)a 
^  <7  COS(^,) 


exp 


(2.3) 
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where  is  the  intensity  of  the  illumination;  ->  is  the  camera  gam;  q  is  a  constant  that  includes  1  \  from 
equation  2.2  and  the  facet  size  (a  variable  m  the  ongmal  Torrance-Sparrow  model);  and  F  is  the  Fresnel 
coefficient  that  describes  what  percentage  of  the  light  is  reflected  at  the  mterface.  Fisa  function  of  geometry . 
wavelength,  polarization  state,  and  index  of  refraction  of  the  matenal  in  question,  however,  this  thesis  will 
follow  the  “Neutral  Interface  Reflection”  model  and  assume  that  F  is  a  constant  for  a  given  type  of  matenal. 


G  is  an  attenuation  factor  that  also  depends  upon  geometry  and  comes  mto  play  at  grazing  angles.  G  is 
a  function  of  incidence  angle,  reflectance  angle,  and  phase  angle  and  is  derived  m  [Torrance  and  Sparrow. 
1967]. 

^  .yrwf,  2cos(e^>cos(er)  2cos(6»,)cos(6',)  ]  ^ 

G  =  - T - , - - - /  (2.4! 

\  cos(6p)  cosidp)  j 

The  phase  angle,  6p,  is  the  angle  between  the  light  source  and  the  camera  with  respect  to  the  object.  Since 
there  is  a  l/cos(0r)  term  in  the  denominator  of  equation  2.3,  that  would  imply  that  the  surface  reflectance 
would  go  to  infinity  as  the  reflectance  angle  approaches  90  degrees.  However,  as  8r  increases  to  90  degrees, 
the  attenuation  term  G  goes  to  zero  faster  than  the  1  /cos  term  goes  to  infinity,  so  that  equation  2.3  does  not 
make  the  improbable  prediction  of  surface  reflection  becoming  infinite.  The  G  term  corrects  for  the  fact  that 
facets  will  tend  to  mask  or  shadow  other  facets  for  grazing  incidence  and  reflectance  angles.  The  1  cos(Br) 
term  predicts  that  the  maximum  of  the  surface  reflection  will  not  always  be  exactly  at  the  perfect  specular 
angle.  This  models  the  “otf-specular  peaks’’  observed  in  real  images  [Torrance  and  Sparrow.  1967], 

A  different  model  for  surface  roughness  is  given  in  [Beckmann  and  Spizzichino,  1963].  In  [Nayar  et  al., 
1991]  this  model  is  compared  with  the  Torrance-Sparrow  model  and  it  is  demonstrated  that  the  two  models 
make  very  similar  predictions  when  the  surface  are  not  extremely  smooth.  A  recent  computer  graphics 
model  also  makes  similar  predictions  about  the  dependence  of  surface  reflectance  upon  surface  roughness 
[He  et  al.,  1991]. 


The  Dichromatic  Reflection  Model  shows  how  the  different  colors  of  body  reflection  and  surface 
reflection  imply  a  planar  distribution  in  color  space.  The  next  chapter  will  show  how  the  different  geometric 
reflection  processes  of  the  two  components  give  rise  to  a  particular  distribution  of  pixels  within  that  plane. 


Chapter  3 


Structure  of  the  Color  Histogram 


When  talking  about  the  color  histogram,  this  thesis  will  mean  a  distribution  of  colors  in  the  three-dimensional 
RGB  space.  Increasingly,  vision  algonthms  are  designed  with  color  images  in  mind  since  color  cameras  are 
becoming  cheaper.  More  importantly,  color  is  a  rich  phenomenon  that  has  the  potential  to  provide  much 
information  about  the  scene.  Traditionally,  histograms  are  associated  with  a  number  of  “bms”  that  cover 
the  range  of  expected  values  and  a  frequency  count  to  record  the  number  of  data  points  that  fall  within  each 
bin.  For  a  typical  imaging  system  with  8  bits  for  each  color  band,  there  are  256^  bins  into  which  a  pixel 
may  fall.  This  work  only  considers  whether  a  bin  is  full  or  empty  in  the  three-dimensional  color  space. 
A  fourth  dimension  could  be  used  to  record  the  number  of  pixels  w'hich  have  a  particular  RGB  value.  Of 
course,  a  fourth  dimension  would  be  difficult  to  visualize.  More  importantly,  this  fourth  dimension  would 
be  dependent  on  such  things  as  image  magnification.  If  the  camera  were  to  zoom  in  on  a  particular  object, 
the  number  of  pixels  it  occupies  in  the  image  would  grow,  causing  a  change  in  the  frequency  count  of  the 
histogram. 

Figure  3.1  contains  a  sketch  of  a  ty-pical  color  histogram  for  a  dielectric  surface  without  interreflection. 
This  simple  case  will  be  studied  first  to  show  what  can  be  learned  from  it  before  proceeding  to  the  more 
complex  analysis  of  interreflection.  As  labeled,  the  histogram  has  two  linear  clusters  of  pixels:  the  body 
reflection  cluster  and  the  highlight  cluster.  The  first  of  those  clusters  extends  from  the  black  comer  of  the 
cube  (at  point  a)  to  the  point  of  maximum  body  reflection  (point  b).  The  other  cluster  starts  somewhere 
along  the  body  reflection  cluster  (point  c)  and  extends  to  the  highlight  maximum  (point  d). 

If  the  object  has  a  completely  black  body  color,  there  will  be  no  body  reflection  component  and  hence 
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Figure  3.1 ;  Histogram  of  a  single  object 

no  body  reflection  cluster.  If  the  body  reflection  color  is  the  same  hue  as  the  surface  reflection  color,  the 
body  reflection  cluster  and  highlight  cluster  will  be  collinear.  This  is  the  case  of  white  and  gray  objects 
since  neither  their  body  color  nor  surface  color  imparts  any  hue  to  reflected  light.  Therefore,  objects  of  these 
colors  cannot  be  analyzed  by  this  type  of  method.  Similarly,  metals  by  their  nature  do  not  exhibit  body 
reflection,  and  so  will  not  have  the  color  distribution  shown  in  figure  3.1. 

This  chapter  examines  the  structure  of  the  color  histogram,  describing  in  detail  the  shape,  orientation, 
and  location  of  both  histogram  clusters.  The  highlighted  points  in  the  diagram  shown  in  figure  3.1  (a,  b. 
etc.)  correspond  to  significant  locations  in  both  the  histogram  and  the  image  and  may  be  used  to  measure 
the  dimensions  of  the  histogram.  The  dimensions  of  the  clusters  relate  in  a  specific,  quantitative  way  to  key 
scene  parameters  including  object  color,  illumination  color,  surface  roughness,  and  phase  angle. 

3.1  The  Body  Reflection  Cluster 

The  linear  cluster  referred  to  as  the  body  reflection  cluster  corresponds  to  pixels  that  exhibit  mostly  body 
reflection  with  very  little  surface  reflection.  If  there  is  no  ambient  illumination  in  the  scene,  this  cluster 
begins  at  the  black  point  in  the  cube  (point  a),  corresponding  to  points  on  the  surface  whose  normal  is 
90  degrees  or  more  away  from  the  direction  of  the  illumination.  Such  points  fall  beyond  the  self-shadow 
line  and  so  are  completely  dark.  The  point  at  the  other  extreme  of  the  body  reflection  cluster  (point  b). 
corresponds  to  the  largest  amount  of  body  reflection  seen  anywhere  on  the  object.  For  Lambertian  objects. 
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pixels  at  this  point  in  the  histogram  correspond  to  points  on  the  surface  with  normals  pouiting  directly  at  the 
light  source.  Assuming  that  the  body  reflection  component  is  Lambertian,  and  recallmg  equation  2.1,  the 
body  reflection  nib  obeys  the  relation 

mb{6i.Br,&p)  =  iBbCosiO,) 

Thus  pixels  located  half-way  along  the  body  reflection  cluster  would  correspond  to  surface  pomts  with 
normals  '2)  or  60  degrees  away  from  the  illummation  direction. 

If  the  object  exhibits  all  possible  surface  normals,  the  body  reflection  cluster  will  be  full  length  and 
densely  filled  with  pixels.  If  the  object  is  composed  of  a  small  number  of  fiat  surfaces,  there  will  be  gaps  in 
the  body  reflection  cluster.  It  is  possible  that  such  an  object  will  not  show  anv  highlight  at  all  even  though 
it  is  very  shiny.  The  sketches  of  histograms  in  this  thesis  assume  that  the  objects  being  observed  exhibit  a 
broad,  continuous  distribution  of  surface  normals.  However,  small  gaps  m  the  body  reflection  cluster  will 
not  harm  this  analysis.  The  only  absolute  requirement  is  that  surface  normals  corresponding  to  points  a  and 
b  must  be  visible  to  the  camera. 

The  sketch  shows  that  the  body  reflection  cluster  is  long  and  narrow.  This  agrees  with  the  simulations 
used  for  this  thesis  and  also  with  real,  high-quality  images  of  clean  surfaces.  Even  for  somewhat  rough 
surfaces,  the  amount  of  surface  reflection  is  very  low  over  all  but  a  very  limited  number  of  reflectance  angles. 
Slightly  thicker  clusters  have  been  observed  for  images  with  greater  amounts  of  noise  and  for  images  of  less 
uniform  surfaces,  but  even  'JO  the  cluster  is  highly  linear. 

A  vector  fitted  to  this  cluster  (from  point  a  to  b)  will  point  in  the  d''5ction  of  the  body  reflection  color 
which  is  the  product  of  the  object  color  and  the  illummation  color.  Automatic  fitting  of  this  vector  has  been 
successfully  demonstrated  [Klinker,  1988].  Once  the  illumination  color  has  been  determined  from  analysis 
of  the  highlight,  the  object  reflectance  may  be  calculated  by  dividing  out  the  influence  of  the  illumination, 
as  proposed  in  some  color  constancy  methods  [D’Zmura  and  Lennie,  1986],  [Healey  and  Bmford,  1987a], 
[Tominaga  and  Wandell,  1989]. 

Assuming  that  there  is  some  point  on  the  object  which  has  a  surface  normal  pointing  directly  at  the 
light  source,  and  which  is  visible  to  the  camera,  then  at  that  point  the  cosine  of  the  incidence  angle  will  be 
one.  This  means  that  the  length  of  the  fitted  vector  (the  magnitude  \alf\  )  corresponds  to  the  gain  times  the 
object’s  apparent  brightness  Bb  If  the  intensity  of  the  illumination  could  also  be  recovered  from  highlight 
analysis,  then  the  albedo  of  the  object  could  be  separated  out  (assuming  that  the  gain  of  the  camera  had  been 
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calibrated).  Otherwise  it  will  be  impossible  to  tell  a  bright  light  shining  on  a  dark  surface  from  a  dim  light 
shining  on  a  light  surface.  Fortunately  highlights  provide  an  invaluable  clue  to  distinguishmg  these  cases. 

To  summarize  this  section; 


•  The  vector  fitted  to  the  body  reflection  cluster  (from  point  a  to  b)  will  point  in  the  direction  of  the 
body  reflection  color,  which  is  the  product  of  the  object  color  and  the  illumination  color. 

•  The  length  of  the  fitted  vector  (the  magnitude  ab  )  corresponds  to  the  gain  times  the  object's  apparent 
brighmess  B),.  If  the  intensity  of  the  illumination  is  recovered  from  highlight  analysis,  then  the  albedo 
of  the  object  Bq  can  be  separated  out  (assuming  that  the  gain  of  the  camera  had  been  calibrated). 


3.2  The  Highlight  Cluster 


The  cluster  of  pixels  called  the  highlight  cluster  corresponds  to  those  pixels  that  show  a  non-negligible 
amount  of  surface  reflection.  This  corresponds  exactly  to  the  area  of  the  image  that  we  would  call  the 
highlight  area.  In  the  histogram,  the  highlight  cluster  starts  where  it  intersects  with  the  body  reflection 
cluster  (point  c),  usually  in  the  upper  half  of  the  body  reflection  cluster.  The  highlight  cluster  extends 
upwards  from  there  to  the  brightest  point  of  the  highlight  (point  d).  For  many  shiny  objects,  the  highlight  is 
so  bright  that  the  highlight  cluster  suffers  color  clipping  and  touches  the  white  point  of  the  color  cube  (an 
RGB  triple  of  [255,  255,  255]  for  an  8  bit  camera)  where  the  highlight  has  saturated  the  camera  [Klinker, 
1988J. 

Recall  from  equation  2.3  that  the  amount  of  surface  reflection  is  given  as 


ms(0n0r-Gp) 


FG{e.,er,e,)c 

c  cos(^,) 


exp 


Obviously  the  magnitude  of  the  highlight  is  dependent  on  a  great  many  things,  including  illumination 
intensity,  imaging  geometry,  and  surface  roughness.  The  next  few  sections  will  describe  how  the  physical 
model  expressed  by  this  equation  leads  to  a  definite  direction,  shape  and  position  for  the  highlight  cluster. 
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3.2.1  Direction  of  Highlight  Cluster 


Surface  reflection  occurs  at  the  interface,  so  it  is  not  affected  by  of  the  body  color  of  inhomogeneous  objects. 
The  Fresnel  coefficient  F  is  very  weakly  dependent  upon  incideu>^e  angle  and  wavelength  of  illumination 
[Shafer,  1^84].  Thus  it  is  often  assumed  to  be  a  constant  for  a  given  type  of  material,  as  in  the  Neutral 
Interface  .Reflectance  (NIR)  model.  Following  this  assumption,  all  the  terms  in  equation  2.3  relate  to  the 
magnitude  of  surface  reflection.  Thus  the  color  of  the  surface  reflection  component  will  be  equal  to  the 
illumination  color. 

It  is  natural  to  assume  that  the  direction  of  the  highlight  cluster  is  related  in  some  w'ay  to  the  surface 
reflection  color.  The  highlight  cluster  is  usually  long  and  narrow  m  shape  and  a  vector  can  be  fitted  to  it  (from 
point  c  to  d),  Klinker  argued  that  tt’e  'ucction  of  this  vector  will  usually  correspond  closely  to  the  surface 
reflection  color  [Klinker,  1985^1  "  us  assumption  is  valid  for  fairly  smooth  objects  where  the  highlight  has 
a  small  area,  and  for  imaffiag  geometries  where  the  body  reflection  changes  slowly  over  that  area.  In  these 
situations,  the  amoiu-i.  of  body  reflection  at  the  base  of  the  highlight  cluster  and  the  amount  at  the  top  of  the 
cluster  varies  bv  only  a  small  amount. 

For  erample,  if  the  camera  and  light  source  are  separated  by  0  degrees  with  respect  to  the  object  (so 
they  are  located  right  on  top  of  each  other  in  space),  the  perfect  specular  angle  will  He  at  0,  =  =  0.  The 

highlight  will  be  spread  around  this  angle.  Right  around  0  degrees,  the  cosine  of  the  incidence  angle  changes 
slowly.  If  the  surface  is  optically  smooth,  the  highlight  will  be  confined  to  angles  close  to  0  degrees,  and 
the  amount  of  body  reflection  varies  only  a  tiny  amount  over  the  entire  area  of  the  highlight.  Since  these 
highlight  pixels  will  all  contain  roughly  the  same  amoimt  of  body  reflection,  any  color  change  within  the 
highlight  can  be  assumed  to  be  due  to  surface  reflection  alone.  Therefore,  for  scenes  where  the  highlight 
occurs  over  a  very  small  range  of  body  reflection  values,  the  assumption  that  the  surface  reflection  color  is 
equal  to  the  highlight  cluster  direction  is  a  good  one. 

On  the  other  hand,  if  the  object  is  optically  rough  and  the  highlight  occurs  on  pan  of  the  object  where  the 
cosine  of  the  incidence  angle  changes  more  rapidly,  then  the  amount  of  body  reflection  can  not  be  assumed 
to  be  constant  over  the  area  of  the  highlight.  The  body  reflection  at  the  base  of  the  highlight  cluster  (point  cl 
may  vary  significantly  from  the  amount  at  the  tip  (point  d).  This  means  that  a  vector  fir  to  the  highlight 
cluster  will  represent  some  change  in  body  reflection  as  well  as  increasing  amotmts  of  surface  reflection, 
causing  the  vector  to  be  skewed  away  from  the  direction  of  the  illumination  color.  This  skew  means  that  the 
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Figure  3.2;  Dependence  of  highlight  cluster  direction  on  object  roughness 


estimate  of  the  illumination  color  made  from  fitting  a  vector  to  the  cluster  will  be  somewhat  inaccurate. 

This  phenomena  can  be  visualized  by  projecting  the  histogram  into  the  plane  defined  by  the  body 
reflection  color  and  surface  reflection  color.  Dichromatic  reflection  is  simulated  for  objects  with  the  same 
body  color  but  with  different  roughness  values.  The  light  source  and  camera  are  20  degrees  apart  with 
respect  to  the  object.  Figtire  3.2  shows  a  cross  section  of  the  histograms  that  result.  The  horizontal  direction 
is  the  direction  of  increasing  amounts  of  body  reflection;  all  three  histograms  fall  right  on  this  line  for  pixels 
where  there  is  no  surface  reflection.  The  vertical  direction  is  defined  by  increasing  amounts  of  surface 
reflection.  For  a  surface  where  the  standard  deviation  of  facet  angles  (roughness)  is  4',  a  vector  fitted  to  the 
highlight  cluster  will  point  exactly  in  this  direction.  A  vector  fitted  to  the  highlight  cluster  when  cr  eqtials  8° 
will  deviate  slightly  from  the  vertical  direction.  In  a  more  extreme  case,  where  cr  equals  16°,  the  vector  will 
deviate  dramatically. 

Figure  3.3  shows  a  sketch  of  this  histogram  for  the  roughest  surface,  with  key  points  labeled.  The 
estimate  of  illumination  color  from  this  highlight  cluster  will  give  the  direction  of  the  light  color  as  cd, 
whereas  the  correct  illumination  color  is  defined  by  the  the  direction  c'd. 

In  the  color  histogram,  the  vectors  describing  the  body  reflection  color  and  the  illiunination  color  are  not 
generally  perpendicular.  The  angle  between  them  depends  upon  the  colors  of  the  tw’o  reflection  components, 
which  are  not  generally  known  in  advance.  If  the  vector  fitted  to  the  highlight  cluster  does  not  point  exactly 
in  the  direction  of  increasing  amounts  of  siuface  reflection,  the  estimate  of  illumination  color  will  be  off  by 


some  amount. 
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Figure  3.3:  A  skewed  highlight 

Furthermore,  the  direction  of  that  vector  is  used  io  calculate  the  amount  of  body  reflection  at  each 
point.  In  [Shafer,  1984]  it  was  proposed  that  an  “intrinsic  image”  of  body  reflection  could  be  generated  by 
determining  the  surface  reflection  vector  and  projecting  all  pixels  along  this  vector  onto  the  body  reflection 
vector.  If  the  fitted  vector  were  skewed  from  the  correct  direction,  pixels  would  be  projected  onto  the  wTOng 
part  of  the  body  reflection  vector.  In  figure  3.2,  all  highlight  pixels  should  be  projected  straight  down  to 
calculate  the  amount  of  body  reflection  present.  However,  for  the  roughest  surface  (cr  =  16),  pixels  would 
end  up  being  projected  at  an  angle,  so  that  the  amount  of  body  reflection  present  would  be  underestimated. 

[Klinker,  1988]  demonstrated  automatic  fitting  of  a  vector  to  highlight  clusters  to  estimate  illumination 
color.  The  vector  fitted  to  the  highlight  cluster  {from  point  c  to  point  d)  is  a  good  first  estimate  of  the 
illumination  color,  but  this  thesis  has  shown  that  it  may  be  skewed.  If  the  surface  roughness  and  the  imaging 
geometry  are  known,  then  the  amoimt  of  skewing  can  be  calculated  and  compensated  for. 

In  summary- 


•  The  vector  fitted  to  the  highlight  cluster  from  point  c  to  d  is  an  estimate  of  the  illumination  color. 


•  That  vector  may  be  skewed  away  from  the  illumination  color,  depending  upon  the  roughness  and 
imaging  geometry. 


The  next  few  sections  will  show  how  the  roughness  and  imaging  geometry  may  be  estimated  directly 
from  the  histogram  itself. 
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3.2.2  Length  of  Highlight  Cluster 


When  looking  at  highlights  on  a  variety  of  surfaces,  it  is  quickly  observ'ed  that  highlights  axe  brighter  and 
sharper  on  some  surfaces,  while  they  are  dimmer  and  more  diffused  on  other  surfaces.  (The  fact  that 
highlights  may  be  diffused  is  a  good  reason  to  avoid  the  term  “diffuse  reflection"  and  use  "body  reflection” 
instead.)  Very  shiny  surfaces  exhibit  only  a  tiny  amount  of  scattermg  of  the  surface  reflection,  whereas  very 
matte  surfaces  have  a  great  deal  of  scattering.  This  scattering  of  surface  reflection  is  a  result  of  the  optical 
roughness  of  the  surface. 

Color  figure  .4.1  show’s  an  image  of  two  billiard  balls  and  the  associated  histogram  for  the  yellow  ball 
on  the  left.  {The  yellow  line  shows  where  the  ball  was  hand  segmented  to  remove  the  number  spot.)  The 
billiard  balls  have  a  very  smooth  finish,  giving  a  shiny  appearance.  Color  figure  .4.2  shows  the  same  balls 
with  the  same  viewing  geometry',  but  here  the  balls  have  been  sprayed  w’ith  an  aerosol.  This  deposits  tiny 
droplets  on  the  surface,  making  it  optically  rough.  Comparing  the  two  histograms,  it  is  immediately  obvious 
that  the  highlight  cluster  for  the  shmy  one  is  considerably  longer  than  the  cluster  for  the  matte  ball. 

It  is  obvious  from  equation  2.3  that  the  sharpness  of  the  peak  is  determined  by  the  standard  deviation 
ex,  and  that  the  height  of  the  peak  is  inversely  proportional  to  ex.  Intuitively  this  makes  sense,  since  surface 
reflection  that  is  scattered  over  a  very  small  area  will  be  more  “concentrated.”  A  smooth  object  w’ill  have 
a  small  standard  deviation  of  facet  slopes,  a,  resulting  in  a  long,  narrow  highlight  cluster.  A  rough  object 
will  have  a  large  cr,  and  so  will  exhibit  a  shorter  and  wider  cluster.  Figure  3.4  shows  a  plot  of  the  length  of 
the  highlight  cluster  vs.  the  object’s  roughness  for  simulated  images  where  all  other  factors  have  been  held 
constant. 

Suppose  that  the  task  is  to  estimate  roughness  values  for  a  variety  of  different  color  objects.  The  body 
reflection  will  be  different  for  each  of  the  surfaces,  but  the  amounts  of  surface  reflection  are  not  affected 
by  this.  The  length  of  the  highlight  cluster  (the  distance  from  c  to  d)  indicates  the  degree  of  roughness, 
regardless  of  body  color. 

According  to  equation  2.3,  it  will  be  necessary  to  know  the  Fresnel  coefficient  for  the  type  of  material 
being  observed  in  order  to  relate  roughness  to  highlight  cluster  length.  However,  it  turns  out  that  for  a  wide 
range  of  plastics  and  paints,  the  indices  of  refraction  are  very  nearly  identical.  For  these  types  of  materials 
the  index  of  refraction  is  typically  between  1 .45  and  1 .56  with  the  majority  between  1 .52  and  1 .55.  Fresnel’s 
law  for  surface  reflection  at  normal  incidence  is  F  =  ((n  ~  l  )/(n  + 1))^  where  n  is  the  index  of  refraction.  The 
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Figxire  3.4:  Dependence  of  highlight  cluster  length  on  object  roughness 

index  of  refraction  is  a  function  of  wavelength,  but  for  these  tv’pes  of  materials,  it  is  nearly  constant  across 
the  visible  range.  Figure  3.5  shows  the  dependence  of  the  length  of  the  highlight  cluster  upon  roughness 
for  different  indices  of  refraction.  The  variation  is  small  enough  to  allow  an  estimate  without  knowing  the 
material,  although  for  an  accurate  measure  the  exact  index  of  refraction  associated  with  the  material  would 
be  needed.  Henceforth  this  thesis  will  assume  that  materials  have  an  mdex  of  refraction  of  1 .5.  corresponding 
to  4.0%  Fresnel  reflectance. 

Figure  3.2  showed  that  if  the  surface  is  very-  roueh  (a  =  16").  the  highlight  cluster  will  be  skewed  from 
the  direction  of  the  illumination  color.  This  makes  the  cluster  longer  than  it  would  otherwise  be.  The  amount 
of  surface  reflection  at  the  bnghtest  point  is  the  vertical  displacement  in  the  graph  (the  distance  from  point 
c'  to  point  d  in  graph  3.3)  rather  than  the  distance  along  the  highlight  cluster  (distance  from  pomt  c  to  point 
d).  For  the  smoother  surfaces  shown  in  the  graph,  the  length  of  the  highlight  cluster  is  virtually  identical  to 
the  amount  of  surface  reflection.  For  very  rough  surfaces,  it  will  be  necessary  to  account  for  the  highlight 
cluster’s  skew  when  calculating  the  length  of  the  highlight  cluster. 

Equation  2.3  also  indicates  that  the  intensity  of  the  light  source  affects  the  magnitude  of  the  surface 
reflection,  and  thus  the  length  of  the  highlight  cluster.  This  is  captured  by  the  term  Bj.  If  all  objects  are 
imaged  under  the  same  light  source,  a  relative  measure  of  roughness  (indicating  how  much  rougher  one 
surface  is  than  another)  could  be  obtained  without  knowing  the  strength  of  the  light.  However,  in  order  to 
calculate  an  absolute  roughness  value,  the  light  source  brightness  must  also  be  analyzed.  It  is  obvious  from 
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Figure  3.5:  Dependence  of  length  on  roughness  for  different  indices  of  refraction  n 

equa  lon  2.3  that  the  length  is  directly  proportional  to  this  brightness.  It  will  be  assumed  that  the  gain  of 
the  camera  7  has  been  calibrated  and  that  the  facet  size  a  is  known.  If  not,  then  only  the  overall  gain  'yoBb 
could  be  recovered  from  the  histogram. 

The  graphs  in  figures  3.4  and  3.5  were  calculated  for  an  imaging  geometiy  vhich  the  light  source  and 
camera  are  separated  by  zero  degrees.  However,  equation  2.3  predicts  that  the  imaging  geometry  will  have 
an  effect  upon  highlight  magnitude,  as  indicated  by  the  cos(0r)  term  in  the  denominator  and  the  attenuation 
term  G  in  the  numerator.  Figure  3  6  shows  how  tne  length  of  the  highlight  cluster  changes  as  the  camera  and 
light  source  are  separated  by  different  phase  angles  with  respect  to  the  object.  It  demonstrates  that  the  length 
changes  slowly  as  the  imaging  geometry  changes.  Figure  3.7  shows  how  the  length  varies  with  roughness 
for  a  variety  of  phase  angles.  Each  curve  represents  a  different  phase  angle.  If  the  imaging  geometry  is  not 
known,  it  would  be  impossible  to  make  fine  distinctions  of  roughness.  It  would  still  be  possible  to  tell  a  very 
smooth  object  (a  =  1  °)  from  a  fairly  rough  one  (cr  =  10°). 

The  result  of  this  section  is  that  the  roughness  of  surfaces  made  from  similar  materials  can  be  directly 
compared  by  examining  their  color  histograms.  This  feature  of  the  histogram  does  not  depend  upon  the 
shape  of  the  object,  and  so  does  not  require  the  fitting  of  surface  onentations  to  image  data.  Once  the 
length  of  the  highlight  cluster  has  been  measured,  it  can  be  used  to  make  an  estimate  of  siuface  roughness. 
Unfortunately,  the  true  length  cannot  always  be  obtained.  For  example,  the  highlight  m  figure  A.l  is  so 
shiny  that  it  saturates  the  camera.  This  means  that  the  highlight  cluster  is  clipped,  so  that  its  true  length  is 
unknown.  As  can  be  seen  from  the  images  in  figures  A.l  and  A.2.  rough  objects  can  be  distinguished  from 
smooth  ones,  but  it  would  not  be  possible  to  distinguish  fairly  smooth  surfaces  from  very  smooth  ones  by 
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Figure  3.6;  Dependence  of  length  on  imaging  geometry 


roughness 


Figure  3.7;  Dependence  of  length  on  roughness  for  different  imaging  geometries 
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this  feamre  of  the  histogram. 

To  summarize  this  section; 

•  The  length  of  the  highlight  cluster  is  inversely  proponional  to  the  roughness  of  the  surface. 

•  The  length  of  the  highlight  cluster  is  directly  proportional  to  the  mtensity  of  the  illumination. 

•  The  length  of  the  highlight  cluster  is  weakly  dependent  upon  the  imaging  geometry. 

•  The  measurement  of  the  highlight  cluster  length  will  be  affected  if  the  estimate  of  illumination  color 
is  skewed. 

3.23  Width  of  Highlight  Cluster 

Another  difference  between  the  histograms  for  the  shiny  and  matte  balls  in  figures  A.  1  and  A.2  is  the  width  of 
the  histogram  at  the  meeting  point  between  the  highlight  cluster  and  the  body  reflection  cluster  (the  distance 
from  point  cl  to  point  c2  in  the  diagram  of  figure  3.1).  The  histogram  of  the  matte  ball  is  wider  where  the 
two  clusters  meet  than  the  histogram  for  the  shiny  ball.  Equation  2.3  shows  that  for  rougher  objects,  with  a 
larger  standard  deviation  of  facet  angles  the  surface  reflection  is  scattered  more  widely. 

The  degree  to  which  the  surface  reflection  is  scattered  may  be  recovered  from  the  width  of  the  highlight 
cluster.  Regardless  of  object  shape,  rougher  surfaces  show  a  non-negligible  amount  of  surface  reflection 
over  a  larger  number  of  reflectance  angles.  In  the  color  histogram,  a  noticeable  amount  of  surface  reflection 
results  in  pixels  that  are  displaced  from  the  body  cluster  in  the  direction  of  the  illumination  color.  By  taking 
any  highlight  pixel  and  projecting  along  the  surface  color  vector  onto  the  body  reflection  vector,  the  amount 
body  reflection  present  in  that  pixel  can  be  calculated.  By  considering  all  the  pixels  in  the  highlight  area 
of  the  image  and  looking  at  how  much  body  reflection  is  in  each  of  them,  some  range  of  body  reflection 
magnitudes  is  obtained.  If  the  surface  is  very  smooth  with  a  sharp,  bright  highlight,  that  range  will  be  small. 
However  for  a  rougher  object  with  a  more  diffused  highlight,  the  range  of  body  reflection  magnimdes  will 
be  larger  since  the  highlight  is  spread  over  a  larger  number  of  surface  normals. 

This  property  is  independent  of  object  size  or  shape.  It  simply  shows  the  variation  in  surface  normals 
over  the  area  of  the  highlight.  It  is  not  necessary  to  fit  a  surface  shape  to  the  image  to  know  how  much 
scattering  the  object  exhibits.  It  is  encoded  right  there  in  the  histogram.  The  only  requirement  of  the  surface 
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Figure  3.8:  Dependence  of  highlight  cluster  width  on  object  roughness 

shape  is  that  the  object  has  surface  normals  corresponding  to  points  cl  and  c2  (see  figure  3.1)  visible  to  the 
camera. 

Objects  with  different  roughness  values  were  simulated;  then  those  pixels  that  showed  a  noticeable 
amount  of  surface  reflection  were  identified.  (“Noticeable”  means  that  the  pixel  was  further  from  the  body 
reflection  cluster  than  the  noise  threshold.)  The  body  reflection  for  each  of  these  points  was  calculated  and 
the  variation  in  value  recorded.  This  variation  was  divided  by  the  overall  length  of  the  body  reflection  vector 
to  yield  a  fraction  (the  length  of  cIq  divided  by  the  length  of  ah  in  figure  3.1).  A  fraction  of  0.5  would  mean 
that  the  highlight  cluster's  base  extended  across  half  the  length  of  the  body  reflection  cluster.  Figure  3.8 
shows  how  the  highlight  cluster  width  varies  with  the  surface  roughness  if  a  point  light  source  is  used. 

Moreover  this  measurement  is  not  hampered  by  camera  saturation  since  body  reflection  generally  does 
not  saturate  the  camera.  It  does  not  depend  on  how  much  surface  reflection  is  at  any  given  point,  only 
whether  there  is  any  at  all.  It  is,  however,  disturbed  by  camera  blooming.  Camera  blooming  means  that 
when  one  pixel  becomes  saturated,  excess  charge  spills  over  into  neighboring  pixels,  making  their  values 
unreliable.  In  cameras  that  are  prone  to  blooming,  a  highlight  bright  enough  to  saturate  the  camera  will  also 
spread  out  beyond  its  true  width.  If  at  all  possible,  a  camera  that  is  resistant  to  blooming  should  be  used  to 
take  pictures  of  objects  with  highlights. 

In  the  case  of  very  rough  surfaces,  the  highlight  cluster  may  be  skewed  from  the  direction  of  the 
illumination  color,  as  shown  in  figure  3.3.  The  width  of  this  cluster  where  it  meets  the  body  reflection 
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cluster  underestimaies  the  variation  of  body  reflectance  values.  The  amount  of  body  reflectance  is  correctly 
determined  by  projecting  all  pixels  along  the  illumination  color  vector  (straight  down  fo’-  this  graph).  The 
correct  measure  of  highlight  width  is  the  variation  in  body  reflectance  (the  distance  from  ci  to  rather  than 
the  distance  from  Ci  toe: }.  However,  the  highlight  and  body  reflection  colors  are  not  generali' perpendicular 
in  most  histograms,  so  just  measuring  the  horizontal  extent  of  the  highlight  cluster  will  not  be  sufficient.  For 
the  smoother  smfaces  shown  in  figure  3.2  (w'hen  cr  <  8“  ),  the  highlight  cluster  is  centered  above  its  base, 
so  the  width  of  the  base  and  the  horizontal  extent  of  the  cluster  are  virmally  identical.  For  these  types  of 
surfaces,  the  width  at  the  base  is  a  very  good  approximation  of  the  overall  width.  However,  for  very  rough 
surfaces,  it  will  be  necessary’  to  account  for  the  skewing  when  determining  the  width  of  the  cluster. 

The  brightness  of  the  illumination  will  also  have  some  effect  on  the  the  width  of  the  highlight  cluster. 
As  the  light  intensity  is  increased,  points  on  the  surface  that  had  amounts  of  surface  reflection  just  under  the 
threshold  may  now  have  amounts  that  exceed  the  threshold.  Clearly  the  width  will  grow  as  the  light  intensity 
grows.  However  the  growth  is  not  linear,  since  width  measures  the  variation  of  body  reflection  values  over 
the  area  of  the  highlight,  not  the  intensity  of  the  highlight  directly.  Figure  3 .9  shows  the  relationship  between 
incident  light  intensity  and  highlight  cluster  width  for  a  fixed  value  of  roughness  and  phase  angle.  The 
horizontal  axis  shows  increasing  light  intensity,  from  50%  to  100%  of  a  theoretical  light’s  maximum  output. 
The  width  grows  very  slowly  over  this  entire  range. 

The  conclusion  is  that  the  width  is  also  affected  by  increasing  illumination  intensity,  but  in  a  different 
way  from  the  effect  of  increasing  intensity  upon  highlight  length  (where  the  growth  is  linear).  This  indicates 
that  there  is  a  difference  between  the  histogram  that  results  from  a  bright  source  illuminating  a  rough  object, 
and  the  histogram  from  a  dim  source  illuminating  a  smooth  object.  Since  the  length  grows  faster  than  the 
width  as  illumination  intensity  is  increased,  this  means  that  the  ratio  between  the  length  and  the  width  w'idth 
will  increase  as  the  illumination  intensity  is  increased. 

Although  the  width  of  the  highlight  cluster  at  its  base  does  not  depend  upon  the  object's  size  and  shape, 
it  does  depend  upon  the  imaging  geometry.  This  is  due  to  the  fact  that  the  width  measurement  indicates  the 
projection  of  highlight  angles  onto  the  body  reflection  vector.  This  effect  is  illustrated  in  figure  3.10. 

The  diagram  on  the  left  of  figure  3.10  shows  a  surface  where  the  highlight  spreads  15  degrees  in  every 
direction  from  its  maximum.  When  the  camera  and  light  source  are  separated  by  a  phase  angle  of  30  degrees 
with  respect  to  the  object,  the  perfect  specular  angle  will  be  at  1 5  degrees  with  respect  to  the  illumination 
vector.  The  highlight  will  spread  over  points  with  surface  normals  ranging  from  0  degrees  to  30  degrees. 
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Figure  3.9;  Dependence  of  width  on  incident  light  intensin- 


Extent  of  highlight 


Figure  3.10:  Illustration  of  why  width  varies  with  imaging  geometry 
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Figure  3.1 1:  Dependence  of  width  on  roughness  for  different  imaging  geomeuies 

(For  the  purpose  of  this  explanation,  ignore  the  influence  of  the  l;cos{6r)  term.)  The  amount  of  body 
reflection  at  these  points  wdll  var>'  from  c<75(0)  =  1 .0  to  coi(30)  =  .87.  a  width  of  0.13. 

If  the  camera  and  light  source  are  separated  by  90  degrees,  the  perfect  specular  angle  will  be  at  45 
degrees  with  respect  to  the  light  source.  This  is  shown  in  the  diagram  on  the  right  of  figtire  3.10.  One  side 
of  the  highlight  will  spread  15  degrees  closer  to  the  direction  of  the  illumination,  so  that  it  reaches  points 
with  surface  normals  of  15  degrees  with  respect  to  the  light.  The  opposite  edge  of  the  highlight  will  spread 
15  degrees  in  the  other  direction,  to  those  points  with  surface  normals  of  45  degrees.  Thus  the  amount  of 
body  reflection  will  vary  from  coj{30)  =  .87  to  cos(60)  =  .50,  a  width  of  0.37. 

Figure  3.1 1  shows  how-  the  width  of  the  highlight  cluster  varies  with  roughness  for  a  variety  of  imaging 
geometries.  The  angle  label  is  a  measure  of  the  phase  angle  (in  degrees)  that  separates  the  light  source  and 
camera  (with  respect  to  the  object).  It  is  assumed  that  the  object  is  small  enough  and  far  away  enough  from 
the  camera  that  this  angle  is  the  same  for  all  points  on  the  surface. 

For  the  case  of  highlight  cluster  width,  the  measurement  is  very  sensitive  to  different  viewing  geometries. 
At  the  same  tune,  the  width  measurement  is  very  i/tsensitive  to  changes  in  illumination  intensity.  The  width 
of  the  highlight  cluster  also  depends  upon  the  degree  of  scattering  of  smface  reflection,  which  is  an  indication 
of  surface  roughness. 


In  summary: 
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•  The  width  is  strongly  dependent  upon  the  roughness  of  the  object. 

•  The  width  is  also  strongly  dependent  upon  the  imaging  geometry'. 

•  The  width  grows  very’  slowly  as  illumination  intensify  is  increased. 

•  The  measurement  of  the  highlight  cluster  width  will  depend  upon  the  estimate  of  illumination  color. 


3.2.4  Intersection  Point  of  Highlight  and  Body  Reflection  Clusters 

When  the  diagram  in  figure  3.1  was  first  introduced,  the  highlight  cluster  was  described  as  beginnmg 
“somewhere”  along  the  body  reflection  cluster.  This  is  rather  vague  and  unsatisfying  so  a  natural  question 
to  ask  w’ould  be,  “Just  where  along  the  body  reflection  cluster  does  it  begin?” 

In  attempting  to  automatically  extract  the  body  reflection  and  highlight  clusters,  Klinker  sought  to 
set  a  limit  upon  where  the  program  should  search  for  the  start  of  the  highlight  cluster.  She  derived  the 
“50%  heuristic”  which  stated  that  for  a  large  range  of  viewing  geometries,  the  highlight  cluster  would  start 
somewhere  in  the  upper  50%  of  the  body  reflection  cluster  [Klinker.  1988]. 

The  distance  along  the  body  reflection  cluster  where  the  two  clusters  meet  (the  length  of  5c  divided  by 
the  length  of  ab  in  figure  3.1)  shows  the  amoxmt  of  body  reflectance  at  those  points  on  the  surface  that  are 
highlighted.  Assuming  that  the  body  reflection  is  Lambertian,  then  equation  2.1  says  the  amount  of  body 
reflection  is  proportional  to  the  cosine  of  the  incidence  angle  9,.  If  the  two  clusters  meet  at  the  maximum 
point  on  the  body  reflection  cluster,  it  means  the  highlight  occurs  at  those  points  that  have  the  maximum 
amount  of  body  reflection,  which  is  precisely  those  points  with  surface  normals  pointing  directly  at  the  light 
source.  If  the  two  clusters  meet  halfway  along  the  body  reflection  cluster,  the  highlight  must  occur  at  points 
with  surface  normal  pointing  cos~ '  ( 1  /2)  or  60  degrees  away  from  the  illumination  direction. 

Since  it  is  assumed  that  the  body  reflection  is  Lambertian,  it  does  not  depend  in  any  way  upon  the 
angle  from  which  it  is  viewed.  Thus  the  body  reflection  does  not  provide  any  information  about  the  camera 
direction.  However,  the  surface  reflection  is  dependent  upon  both  the  illumination  and  camera  directions. 
Neglecting  for  the  moment  the  1  /cos(^r)  term  in  equation  2.3,  it  is  easy  to  see  that  the  maximum  amount  of 
surface  reflection  will  occur  approximately  at  those  points  on  the  surface  where  the  angle  of  incidence  equals 
the  angle  of  reflection.  Thus  if  the  highlight  occurs  at  a  point  where  the  surface  normal  faces  10  degrees 


52 


CHAPTER  3.  STRUCTURE  OF  THE  COLOR  HISTOGRAM 


C 

*5 

s 

I 

Vi 


80.00 

60.00 

40.00 

20.00 

0.00. 

-20.00 


-  angle  =  0 

.  angle  =  20 

—  -  —  angle  =  40 

—  ■  ■  —  angle  =  60 

—  -  angle  =  80 


_ I _ I 

0J20  0.40 


060  060  LOO 

body  reflection 


FigTire  3.12:  Dlustraiion  of  how  highlight  cluster  varies  with  changing  imaging  geometry 


away  from  the  light  source  direction,  the  light  source  and  camera  must  be  20  degrees  apan  with  respect  to 
that  point  on  the  surface. 

Figure  3.12  graphically  illustrates  this  phenomenon.  Once  again  the  histograms  •’ave  been  projected 
into  the  plane  defined  by  the  body  reflection  and  surface  reflection  colors,  so  that  the  hori2X)ntal  direction 
indicates  inaeasing  amounts  of  body  reflection  and  the  vertical  direction  indicates  increasing  amounts  of 
siuface  reflection.  This  time,  the  amoxmt  of  roughness  has  been  held  constant  at  cr  =  1  “  while  the  phase 
angle  between  the  light  source  and  camera  has  been  varied  from  0  to  80  degrees.  This  graph  shows  how  the 
intersection  point  (\ac\/lab\)  decreases  as  the  phase  angle  between  the  camera  and  light  source  inaeases. 
Incidentally  it  also  shows  how  the  length  and  width  of  the  cluster  are  affected  by  imaging  geometry  as 
described  in  sections  3.2.2  and  3.2.3. 

It  does  not  matter  whether  the  object  has  one  highlight  or  many.  The  highlight  or  highlights  will  always 
occur  at  points  with  the  same  surface  normal  for  a  given  imaging  geometry,  and  that  is  what  determines 
the  intersection  point  of  the  clusters.  Figure  3.13  graphs  intersection  point  vs.  imaging  geometry.  The 
horizontal  axis  shows  the  phase  angle  between  the  light  source  and  the  camera  with  respect  to  the  object.  A 
negative  number  means  that  the  light  source  is  to  the  left  of  the  camera,  whereas  a  positive  number  means 
the  light  source  is  to  the  right  of  the  camera.  The  vertical  axis  is  the  ratio  ]ac\/ \ab\ .  As  can  be  seen  from  the 
graph,  the  meeting  point  is  never  less  than  1/2  of  the  way  along  the  body  reflection  vector,  showing  that  the 
50%  heuristic  is  a  sound  one. 

The  previous  few  paragraphs  assumed  that  the  intersection  point  of  the  two  clusters  shows  the  amount  of 
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phase  angle  between  light  and  camera 

Figure  3.13:  Dependence  of  intersection  point  on  imaging  geometry 


body  reflectance  at  the  brightest  point  in  the  highlight.  This  assumption  is  valid  for  cases  where  the  highlight 
cluster  extends  in  approximately  the  same  direction  as  the  illumination  color.  However  if  the  highlight 
cluster  is  significantly  skewed  away  from  the  direction  of  the  illumination  color,  the  intersection  point  will 
give  an  incorrect  estimate  of  the  imaging  geometry.  This  is  the  case  for  the  roughest  surface  (cr  =  16°) 
shown  in  figure  3.2.  All  three  histograms  correspond  to  a  phase  angle  of  20  degrees,  and  indeed  the  highest 
point  of  each  highlight  cluster  occurs  at  the  same  horizontal  coordinate  (which  indicates  the  amount  of  body- 
reflection).  However,  for  the  rough  surface  shown  m  figure  3.3,  the  correct  intersection  ratio  is  given  by 
\ac'\/\ab\  rather  than  \ac\j\ab\.  This  skewing  of  the  highlight  cluster  means  that  the  corrected  estimate  of 
illumination  color  is  needed  to  correctly  calculate  imaging  geometry  for  very  rough  surfaces. 

The  1  jcosiOr)  term  in  the  surface  reflection  equation  means  that  the  maximum  amount  of  surface 
reflection  will  not  always  occur  precisely  at  the  perfect  specular  angle.  This  is  particularly  true  of  rougher 
surfaces  where  the  highlight  is  spread  over  a  wide  range  of  reflectance  angles  so  that  1  lcos{6r)  varies  enough 
to  make  a  difference.  This  gives  rise  to  “off-specular  peaks”  described  in  ^Torrance  and  Sparrow,  1967]  and 
[Nayar  et  al.,  1991].  Figure  3.14  shows  how  the  intersection  graph  changes  for  different  roughness  values. 
The  graphs  shows  that  the  roughness  does  indeed  affect  the  intersection  point  of  the  two  clusters,  but  only 
by  a  small  amount  for  the  most  conunon  imaging  geometries  (light  source  and  camera  separated  by  less  than 
45  degrees). 

The  relationship  described  here  between  the  intersection  of  the  two  clusters  and  the  position  of  the  light 
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Figxire  3.14;  Dependence  of  intersection  on  geometry  for  different  roughness  values 


source  assumes  that  surface  normals  corresponding  to  key  points  in  the  histogram  (a,  b,  and  d  in  figure  3.1) 
are  visible  to  the  camera.  In  particular,  the  requirement  that  point  b  is  visible  means  that  the  phase  angle 
must  be  less  than  or  equal  to  ninety  degrees.  Otherwise  the  point  of  maximum  highlight  would  occur  at 
surface  normals  out  of  view  of  the  camera.  While  situations  where  the  light  is  further  than  90®  from  the 
camera  can  occur,  a  large  number  of  the  surface  normals  that  would  otherwise  be  visible  to  the  camera  will 
be  dark,  since  they  will  be  beyond  the  self  shadow  line.  This  imaging  geometry  is  not  common  in  scenes 
where  the  object  is  deliberately  illuminated  for  the  picture,  since  it  is  customary  to  illuminate  scenes  so  a 
large  number  of  surface  normals  will  be  visible. 

This  method  for  determining  the  phase  angle  between  the  light  and  camera  by  examining  the  highlight 
cluster’s  intersection  with  the  body  reflection  cluster  is  implicitly  photometric,  since  it  relies  on  the  amount 
of  body  reflection  at  certain  points.  However,  it  does  not  require  the  fitting  of  surface  orientations  to  image 
data,  nor  does  it  require  knowledge  of  the  albedo  of  the  surface.  Moreover,  it  is  done  from  a  single  image. 

To  summarize  this  section; 


•  The  intersection  of  the  two  clusters  is  strongly  dependent  upon  the  imaging  geometry. 


•  The  intersection  measurement  is  weakly  dependent  upon  imaging  geometry. 


•  The  intersection  measurement  will  be  decreased  if  the  estimate  of  illumination  color  is  skewed. 
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Figure  3.15;  Dependence  of  histogram  features  upon  scene  parameters 

Recently  Wolff  has  shown  that  the  Lambertian  model  is  somewhat  inaccurate  for  body  reflection  from 
many  real  surfaces,  paxticularly  at  extreme  reflectance  angles;  he  has  proposed  a  different  model  to  replace 
it  [Wolff,  1992],  Also,  some  researchers  prefer  to  use  the  Beckmann-Spizzichino  model  [Beckmaim  and 
Spizzichino,  1 963],  or  a  combination  of  the  Beckmann-Spizzichino  and  the  Torrance-Sparrow  models  [Nayar 
et  al.,  1991],  to  describe  surface  scattering.  While  a  choice  of  different  models  would  change  the  exact 
positions  of  the  graphs  in  this  section,  the  general  relationships  would  remain  the  same.  These  dependencies 
of  the  color  histogram  upon  scene  parameters  are  a  consequence  of  the  different  geometric  distributions  as 
well  as  the  different  hues  of  the  two  components  of  reflection. 

Sections  3.2.2  and  3.2.3  showed  how  the  roughness  of  the  object  affects  the  from  the  length  and  width 
of  the  highlight  cluster;  however  they  also  showed  that  there  is  some  dependence  on  imaging  geometry. 
Now  this  section  shows  how  the  imaging  geometry  affects  from  the  intersection  point  of  the  two  clusters, 
but  that  there  is  some  dependence  upon  roughness.  The  degree  of  dependence  of  each  of  these  histogram 
measurements  upon  the  scene  parameters  is  characterized  in  figure  3.15. 

Furthermore,  in  section  3.2.1  it  was  noted  that  the  direction  of  the  highlight  cluster  could  be  skewed 
away  from  the  direction  of  the  illumination  color,  depending  upon  the  amoxmt  of  roughness  and  the  imaging 
geometry.  Yet  the  estimates  of  cluster  length,  width,  and  intersection  point  depend  to  a  certain  extent  upon 
knowing  the  true  direction  of  the  illumination  color.  Obviously  these  factors  are  all  interdependent.  One 
solution  might  be  to  first  estimate  those  things  that  are  weakly  dependent  on  unknowns  and  then  iterate. 
However  this  thesis  has  chosen  to  solve  for  all  values  simultaneously.  That  solution  will  be  developed  in 
chapter  5. 


56 


CHAPTER  3.  STRUCTURE  OF  THE  COLOR  HISTOGRAM 


Chapter  4 


Color  Histogram  Structure  n  hen 
Interreflection  is  Present 


In  roany  models  of  image  formation,  the  illumination  is  considered  to  have  a  ver>  simple  form:  either  the 
illumination  is  invariant  with  geometry  (modeling  "'imbient''  lighting  )  or  is  non-zero  in  only  one  direction 
(modeling  a  ‘'point  source").  In  both  these  cases  the  illumination  is  considered  to  have  only  cne  color  so 
that  algorithms  like  color  constancy  may  be  applied.  The  usual  model  allows  only  "primary  reflection,"  in 
that  the  light  is  assumed  to  hit  objects  in  the  scene  once  before  entering  the  camera. 

Unfortunately  this  type  of  model  is  not  sufficient  to  explain  interreflection,  which  can  occur  even  in 
simple  images.  Therefore,  it  would  be  advantageous  to  use  a  more  general  model  which  allows  for  reflection 
of  light  between  objects.  In  order  to  do  this,  this  thesis  will  use  an  expanded  version  of  the  dichromatic 
reflection  model. 


4.1  Modeling  Interreflection 

As  is  done  with  the  traditional  dichromatic  reflection  model,  ihe  reflection  from  an  object  is  separated  into 
body  reflection  and  surface  reflection.  Consider  what  happens  when  each  of  the  two  components  from  one 
surface  (A)  is  reflected  by  another  surface  (B)  in  two  different  ways.  This  gives  rise  to  “secondary  reflection" 
with  four  components  whose  magnitudes  are: 
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body  reflection  of  A  times  the  body  reflection  of  B 
body  of  A  times  surface  of  B 
surface  of  A  times  body  of  B 
surface  of  A  times  surface  of  B 

If  these  are  taken  together  with  the  primary  reflection  components  >  they  make  a  total  of  six  reflection 
components  [Shafer  et  al.,  1990],  [Bajcsy  etal.,  1990]. 

In  the  “tertiary  reflection”  case,  the  light  makes  three  bounces  before  entering  the  camera,  gi%’ijig  2 '  or  8 
possible  reflection  components.  Taken  together  with  the  primary  and  secondary  reflection  components,  there 
are  a  total  of  14  reflection  components.  In  general,  for  n-ary  reflection,  there  are  2'^  reflection  components 
with  exactly  n  bounces,  for  a  total  of  2""^'  -  2  reflection  components  with  up  to  n  bounces.  Fortunately, 
objects  usually  reflect  a  fraction  of  the  light  impinging  upon  them,  so  that  in  most  cases  the  high-order 
reflection  components  will  approach  zero  for  large  n. 

Color  figure  A,3  shows  the  results  of  simulating  interreflection  for  two  cylinders  that  are  identical  except 
for  hue.  The  Torrance-Sparrow  model  was  used  to  simulate  the  surface  reflectance,  with  a  roughness  of 
cr  =  1®.  In  this  simulation  the  light  source  and  camera  are  at  the  same  position  in  space  and  are  infinitely  far 
from  the  objects.  The  top  portion  of  the  picture  shows  the  results  when  only  primary  reflection  is  allowed, 
so  there  is  no  interreflection.  The  middle  section  of  the  picture  shows  simulated  primary  and  secondary 
reflection  only.  The  bottom  section  shows  a  simulation  of  primary,  secondary,  and  tertiary  reflections. 
The  difference  caused  by  the  addition  of  tertiary  reflection  is  small  and  not  noticeable  to  the  human  eye. 
Henceforth  this  thesis  will  only  consider  primary  and  secondary  reflection  for  this  type  of  surface. 

Figure  4.1  shows  a  graph  of  the  magnitudes  of  the  6  reflection  components  for  the  simulated  cylinder 
A  shown  in  figure  A.3.  Each  of  the  components  represents  a  distinct  path  from  the  light  source  to  the 
camera.  The  horizontal  axis  is  the  angular  position  of  points  on  the  cylinder,  with  zero  corresponding  to 
the  points  with  surface  normals  pointing  at  the  camera,  and  negative  angles  corresponding  to  the  side  facmg 
the  other  cylinder.  The  vertical  axis  inu.  'ates  the  relative  strength  of  the  component  at  that  position  on  the 
cylinder.  Notice  that  although  there  is  no  body  reflection  from  cylinder  A  past  -90  degrees  (this  is  beyond 
the  self-shadow  line),  some  of  the  interreflection  components  are  positive  at  -100  degrees  since  these  points 
can  “see”  light  reflected  from  cylinder  B. 
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Although  this  model  considers  6  different  paths  by  which  light  may  reach  the  camera,  the  components 
will  show  at  most  four  different  hues.  This  is  because  the  Neutral  Interface  Reflection  model  predicts  that 
light  that  bounces  from  the  surface  of  A  will  have  the  same  hue  as  the  illumination  color;  thus  will 
have  the  same  hue  as  while  will  have  the  same  hue  as  Figure  4.2  shows  a  graph  of  the 
magnitudes  of  the  4  distinct  hues  obtained  when  components  with  the  same  hue  are  combined. 

Figure  4.3  shows  a  sketch  of  the  color  histogram  that  arises  from  a  single  object  with  interreflection. 
The  relevant  parts  are  labeled.  The  body  reflection  cluster  and  highlight  cluster  are  in  the  same  positions  as 
they  were  for  the  simpler  histogram.  However  the  highlight  cluster  has  been  renamed  the  primary  highlight 
cluster.  This  is  because  there  is  now  a  distinct  highlight  cluster  called  the  secondary  highlight  cluster.  The 
interreflection  cluster  refers  to  the  remaining  clump  of  pixels  that  deviate  from  the  body  reflection  near  its 
base.  The  next  few  sections  will  examine  in  detail  these  new  histogram  features. 


4.2  The  Secondary  Highlight  Cluster 


The  secondary  highlight  r-i  '.ster  results  from  interreflection,  specifically  the  surface-to-surface  component. 
It  corresponds  to  those  pixels  that  foim  a  secondary  highlight,  a  mirror  image  of  the  highlight  on  the  other 
object.  Although  the  secondary  highlight  is  quite  small,  it  is  visible  in  the  photo  of  the  shiny  billiard  balls 
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Figure  4.3;  Histogram  for  inteireflection 
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in  figure  A.l. 

However,  the  secondarv-  highlight  cluster  is  absent  from  the  histogram  corresponding  to  the  matte  balls 
in  figure  A.2.  Accordingly,  a  secondary  highlight  is  not  discernible  in  the  original  image. 


4.2.1  Length  of  the  Secondary  Highlight  Cluster 

The  billiard  balls  in  figures  A.l  and  A.2  demonstrate  that  the  secondary  highlight  depends  upon  the  surface 
roughness.  A  primary  highlight  on  surface  A  has  a  brightness  related  to  its  roughness  parameter  c  as 
explained  in  section  3.2.2.  When  a  second  surface  B  reflects  a  mirror  image  of  this  primary  highlight,  the 
brightness  of  the  resulting  secondary  highlight  must  vary-  as  the  brightness  of  the  primary  highlight.  If  A's 
primary  highlight  is  dimmer,  thenfi's  secondary  highlight  must  be  dimmer  also.  Furthermore,  the  brighmess 
of  the  secondary  highlight  on  S  also  depends  upon  the  reflectance  characteristics  of  B.  If  the  surface  of  B 
is  optically  rough,  then  it  will  scatter  light  that  impinges  upon  it.  This  applies  to  light  that  comes  directly 
from  the  light  source,  but  also  to  light  that  is  reflected  from  other  objects.  As  B  gets  rougher,  the  secondary- 
highlight  will  become  more  scattered  and  hence  less  bright. 

The  brighmess  of  the  secondary  highlight  determines  the  length  of  the  secondary  highlight  cluster.  This 
is  because  larger  amounts  of  surface-to-surface  reflection  result  in  pixels  displaced  further  from  the  body 
reflection  cluster.  This  length  (the  distance  from  points  c  to  f  in  figure  4.3)  depends  upon  the  roughness 
of  both  surfaces.  If  both  surfaces  are  optically  rough,  the  secondary  highlight  may  be  completely  missing, 
resulting  in  a  length  of  0.  If  both  surfaces  are  smooth  as  in  figure  A.l,  the  secondary  highlight  cluster  will 
have  a  substantial  length. 

Notice  however  that  the  secondary  highlight  cluster  in  figure  A.l  does  not  reach  the  white  point  of 
the  cube,  indicating  that  the  secondary  highlight  does  not  saturate  the  camera.  In  this  picture  the  primary 
highlight  did  saturate  the  camera,  so  its  true  length  cannot  be  measured  from  the  histogram.  But  when  the 
surface  reflection  is  reflected  a  second  time,  only  a  fraction  of  the  incoming  light  is  reflected.  Even  though 
the  primary  highlight  is  too  strong  for  the  camera,  the  secondary  highlight  is  within  its  dynamic  range.  Thus 
the  secondary  highlight  provides  a  second  constraint  upon  the  roughness  of  the  object,  with  the  benefit  that 
it  is  much  less  likely  to  saturate  the  camera. 


62 


CHAPTER  4.  COLOR  HISTOGRAM  STRUCTURE  WHEN  L\TERREFLECT/ON  IS  PRESEhT 


4.2.2  Direction  of  the  Secondary  Highlight  Cluster 

The  secondary  highlight  cluster  grows  upwards  from  the  body  reflection  cluster  as  the  amount  of  surface-to- 
stirface  reflection  increases.  This  is  similar  to  the  case  for  the  prunaiy'  highlight  cluster,  which  grows  upward 
as  the  amount  of  surface  reflection  mcreases.  Assuming  that  reflection  at  the  mt  erf  ace  does  not  affect  the 
color  of  reflected  light,  then  surface-to-surface  reflection  should  have  the  same  color  as  surface  reflection. 

However,  after  examining  the  histogram  in  figure  A.l,  the  direction  of  the  secondaiy  highlight  cluster 
appears  to  be  different  from  that  of  the  pnmary  highlight  cluster.  Although  the  direction  of  these  clusters  is 
mostly  determined  by  the  direction  of  mcreasing  illumination  color,  it  is  also  affected  by  the  amount  of  color 
change  in  the  other  reflection  components  over  the  area  for  which  the  highlight  occurs.  Section  3.2.1  showed 
that  for  relatively  smooth  surfaces,  the  amount  of  body  reflection  changes  slowly  over  the  area  of  the  pnmary 
highlight,  so  that  the  direction  of  the  associated  cluster  is  very  nearly  the  direction  of  the  illumination  color. 

The  same  claim  cannot  be  made  for  the  secondaiy  highlight.  The  region  of  the  object  where  it  tends 
to  occur  is  often  experiencing  a  rapid  change  in  body  reflection.  Furthermore  it  may  also  be  experiencing 
a  rapid  change  in  the  amount  of  other  interreflection  components.  In  figures  4.1  and  4.2.  the  secondary 
highlight  is  centered  around  -45  degrees.  In  this  region  of  the  cylinder,  the  body  color  of  cylinder  A  is 
changing  faster  than  it  is  in  the  area  of  the  primary'  highlight  (centered  aroimd  0  degrees).  Furthermore,  the 
body-to-body  and  surface-to-body  components  are  dropping  off  in  this  region.  The  changes  in  these  color 
components  affect  the  direction  of  the  secondary  highlight  cluster.  Thus  a  vector  fitted  to  the  secondary 
highlight  cluster  may  deviate  even  more  from  the  illumination  color  than  the  vector  fitted  to  the  primary 
highlight  cluster. 


4.23  Intersection  Point  of  the  Secondary  Highlight  Cluster 

The  location  of  a  secondary  highlight  on  an  object  depends  upon  the  relative  positions  of  the  objects  and  upon 
the  positions  of  the  light  source  and  camera.  For  surfaces  that  exhibit  only  perfect  mirror-like  reflection,  a 
secondary  highlight  onB  occurs  only  where  the  angles  of  incidence  and  reflectance  are  equal  on  object  A, 
the  angles  of  incidence  and  reflectance  are  equal  on  B  and  the  reflectance  ray  from  A  is  coincident  with  the 
incidence  ray  on  5.  The  diagram  in  figure  4.4  shows  the  angles  for  which  this  occurs  when  two  cylinders  are 
aligned  so  the  line  connecting  them  is  perpendicular  to  the  optical  axis  of  the  camera,  and  the  camera  and 
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Figure  4.4;  Location  of  a  secondary’  highlight 


light  source  are  at  iofinitv’. 

If  the  light  source  is  moved  so  that  it  is  separated  from  the  camera  by  a  non-zero  phase  angle  relative  to 
the  objects,  the  angles  for  which  perfect  specular  reflection  occurs  will  change,  This  is  also  the  case  when 
the  cylinders  are  moved  relative  to  each  other.  Furthermore  because  of  off-specular  peaks,  the  brightest 
point  of  the  secondary  highlight  will  not  always  be  at  the  perfect  specular  angle. 

Thus,  it  is  surprising  to  And  that  the  position  of  the  secondary  highlight  on  the  cylinder  actually  changes 
very  little  as  the  light  source  and  camera  are  separated  by  increasing  angles.  Figure  4.5  shows  the  change 
in  the  location  of  the  secondary  highlight  for  the  simulated  cylinders  as  the  light  source  is  moved  relative  to 
the  camera.  The  cylinders  remain  aligned  such  that  the  line  connecting  their  center  is  perpendicular  to  the 
optical  axis  of  the  camera.  The  horizontal  axis  of  the  graphs  shows  the  phase  angle  between  the  camera  and 
light  source,  with  negative  numbers  indicating  that  the  light  source  is  to  the  left  of  the  camera  when  facing 
the  objects.  The  vertical  axis  is  the  reflectance  angle  on  the  cylinder  at  which  the  secondary  highlight  has 
its  peak.  Notice  that  when  the  camera  and  light  source  are  separated  by  zero  degrees,  the  peak  occurs  at  ^5 
degrees  as  shown  in  figure  4.4.  Over  the  whole  range  of  phase  angles  shown,  the  secondary  highlight  only 
moves  from  -41  degrees  to  -54  degrees. 

However,  examining  the  point  in  the  histogram  where  the  secondary  highlight  cluster  meets  the  body 
reflection  cluster  shows  that  this  point  is  highly  variable.  Figure  4.6  demonstrates  how  the  intersection  point 
varies  with  imaging  geometry.  The  vertical  axis  is  the  distance  along  the  body  reflection  vector  as  a  fraction 
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phase  angle  between  light  and  camera 


Figure  4.5;  Angle  at  which  secondary  highlight  occurs 


of  its  total  length  (the  length  of  ae  divided  by  the  length  of  ab  in  figure  4.3). 

Even  though  the  secondary  highlight  moves  very  little  as  the  light  is  moved  around,  the  body  reflection 
is  directly  following  the  light  source.  When  the  light  is  45  degrees  to  the  left  of  the  camera  (at  -45),  the 
secondary  highlight  is  actually  at  the  maximum  point  of  body  reflection  (100%).  Looking  back  to  figure  3.13. 
it  can  be  seen  that  the  primary  highlight’s  intersection  point  for  that  same  imaging  geometry  is  at  927c  of 
the  maximum  body  reflection.  This  means  that  the  primary  and  secondary  highlight  clusters  have  “crossed 
over”  in  the  histogram,  with  the  secondary  highlight  cluster  actually  intersecting  the  body  reflection  cluster 
above  the  primary  highlight  cluster.  The  histogram  for  a  simulated  red  cylinder  in  this  imaging  situation  is 
graphed  in  figure  4.7.  (Notice  that  the  primary  highlight  cluster  is  bent  when  it  reaches  the  camera  maxiiruim 
of  255.) 

As  with  the  primary  highlight  cluster,  the  intersection  point  of  the  secondary  highlight  cluster  also 
depends  on  the  surface  roughness.  If  the  highlight  is  scattered  over  a  wider  range  of  reflectance  angles,  the 
1  jcos{6r)  term  varies  more  across  the  highlight,  causing  off-specular  peaks.  Just  as  off-specular  peaks  affect 
the  intersection  of  the  primary  highlight  cluster  and  body  reflection  cluster,  they  also  change  the  intersection 
of  the  secondary  highlight  cluster.  This  dependence  is  extremely  small,  as  shown  in  figure  4.8.  For  ver\- 
rough  surfaces,  the  secondary  highlight  will  probably  be  absent  altogether. 

Assuming  the  secondary  highlight  cluster  can  be  found  in  the  histogram,  its  intersection  point  may  give  a 
ir  ore  precise  estimate  of  imaging  geometry  than  the  primary  highlight  cluster,  since  the  variation  with  angle 
is  greater.  However,  there  is  an  ambiguity  centered  about  -45  degrees.  Moreover  the  graphs  in  figures  4.6 
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phase  angle  between  light  and  camera 

Figure  4.6:  Dependence  of  secondarj’  highlight  intersection  point  upon  geometry 


Figure  4.7:  Histogram  where  primary  and  secondary  highlight  clusters  have  crossed  over 
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angle  between  light  and  camera 

Figure  4.8:  Dependence  of  intersection  upon  geometry  for  different  roughness  values 


and  4.8  only  apply  to  situations  where  the  cylinders  are  aligned  as  previously  described.  For  more  general 
viewing  situations  and  object  shapes,  the  graphs  may  be  different.  Fortunately,  it  is  already  possible  to  make 
an  estimate  of  light  position  from  the  graph  in  figure  3.14  for  the  primary  highlight  cluster.  The  position  of 
the  secondary  highlight  will  allow  a  resolution  of  that  graph’s  ambiguity  about  zero.  The  secondary  highlight 
will  occur  on  a  side  of  object  A  that  faces  the  other  object  B.  If  the  secondary  highlight  cluster  intersects  the 
body  reflection  cluster  in  the  upper  region,  that  implies  the  light  is  also  on  that  same  side.  Conversely,  if  the 
secondary  highlight  intersection  occurs  in  the  lower  portion  of  the  body  reflection  cluster,  that  indicates  that 
the  light  is  on  the  opposite  side. 

If  the  primary  and  secondary  highlight  clusters  can  change  places  in  the  histogram,  it  will  be  necessary 
to  find  a  way  to  distinguish  them.  Since  the  secondary  highlight  cluster  will  be  shorter  than  the  primary 
one,  the  length  can  be  used  to  tell  them  apart.  Somewhere  around  -30  degrees  the  graphs  show  that  both 
clusters  have  an  intersection  point  at  97 %  of  the  length  of  the  body  reflection  cluster.  If  the  direction  of  these 
two  clusters  is  also  the  same,  they  will  be  indistinguishable.  Also  if  both  surfaces  are  matte,  the  secondary 
highlight  cluster  will  be  completely  absent. 
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4.3  The  Interreflection  Cluster 

The  interreflection  cluster  refers  to  those  pixels  that  fall  outside  the  dichromatic  plane  defined  by  the  body 
reflection  vector  and  illumination  vector.  These  pixels  are  precisely  those  that  experience  the  hue  shift 
seen  in  interreflection.  They  cause  considerable  trouble  for  se?-'ientation  algorithms  since  they  cannot  be 
explained  by  the  regtilar  dichromatic  reflection  model.  In  figures  A.l  and  A.2,  some  of  the  pixels  from  the 
yellow  ball  have  the  same  color  as  pixels  from  the  red  ball,  so  they  could  not  be  correctly  segmented  based 
on  hue. 

Still,  some  things  can  be  said  about  the  interreflection  cluster.  If  the  secondary  highlight  cluster  is  caused 
bv  surface-to-surface  reflection,  the  interreflection  cluster  must  result  from  some  combination  of  the  other 
three  types  of  interreflection;  body-to-surface,  surface-to-body,  and  body-to-body  reflection. 

43.1  Surface  to  Body  Reflection 

The  surface-to-body  reflection  component  of  interreflection  causes  few  difficulties  for  computer  vision 
algorithms.  Assuming  that  surface  reflection  does  not  color  the  illumination,  then  the  illumination  from  the 
surface  of  an  object  A  that  is  reflected  by  the  body  of  object  B  will  have  the  same  hue  as  light  that  comes 
straight  from  the  light  source  to  the  body  of  B.  The  object  A  acts  a  duplicate  light  source  next  to  B.  Any 
pixels  that  have  a  significant  surface-to-body  component  will  merely  be  moved  further  up  the  body  reflection 
cluster  since  they  have  the  same  hue  but  are  brighter.  For  example,  points  beyond  the  self  shadow  line  of  B 
would  normally  be  at  the  black  point  of  the  color  cube.  If  they  face  a  lit  side  of  object  A  they  will  have  a 
small  amount  of  the  body  reflection  color  of  B,  moving  them  further  up  the  body  reflection  cluster  for  B. 

43.2  Body  to  Body  Reflection 

The  body-to-body  reflection  component  of  interreflection  is  the  one  that  has  most  often  been  explored  by- 
vision  researchers  [Horn,  1977],  [Forsyth  and  Zisseiman,  1989],  fNayarera/.,  1990],  [Jang,  1991],  [Funt 
and  Drew,  1991].  They  all  make  the  assumption  that  the  surfaces  they  are  dealing  with  are  completely  matte 
and  so  ignore  the  other  three  components  of  interreflection.  The  work  m  [Bajesy  et  al.,  1989]  works  on 
objects  with  highlights,  but  where  the  interreflection  has  a  small  enough  effect  on  hue  to  be  segmented  along 
with  non-interrefiected  pixels. 
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The  body-to-body  reflection  color  is  the  product  of  three  spectral  functions;  the  illumination  color,  the 
body  color  of  the  first  object,  and  the  body  color  of  the  second  object.  This  multiplication  takes  place  at  each 
wavelength  within  the  visible  spectrum.  Suppose  for  discussion  purposes  that  the  color  of  the  illumination 
is  white  so  that  its  influence  can  be  ignored.  Then  the  spectral  signal  resulting  from  body-to-body  reflection 
color  will  be  high  only  for  those  portions  of  the  spectrum  where  both  body  colors  are  high.  The  resulting 
spectral  signal  will  be  referred  to  here  as  the  “overlap  color”  since  it  reflects  light  only  where  the  spectra  of 
the  two  original  colors  overlap. 

To  illustrate  the  effects  of  color  overlap,  this  section  will  examine  the  effects  for  four  different  colors 
of  objects:  red,  yellow,  blue  and  purple.  The  spectral  reflectance  curves  for  these  objects  are  shown  in 
figure  4.9.  The  colors  associated  with  these  curves  look  snong,  or  saturated,  with  very  little  added  white. 
The  fact  that  each  of  the  curves  is  near  zero  for  some  portion  of  the  spectrum  means  that  no  white  could 
be  subtracted  out  to  make  the  color  look  more  pine.  These  spectral  signals  are  turned  into  RGB  values 
by  multiplying  them  by  the  camera’s  spectral  responsiveness  in  each  of  the  three  color  bands  and  then 
integrating. 

For  the  case  of  body-to-body  reflection  between  a  red  object  and  a  blue  one,  the  color  overlap  is  nerr 
zero.  This  is  because  the  spectral  reflectance  of  the  red  object  is  high  in  the  longer  wavelengths  where  the 
blue  reflectance  is  low,  and  the  blue  reflectance  is  high  in  the  shorter  wavelengths  where  the  red  reflectance 
is  low.  There  is  no  part  of  the  spectrum  for  which  both  colors  reflect  appreciable  amoimfs  of  light.  Thus  the 
body-to-body  reflection  comptonent  of  interreflection  between  two  objects  of  these  colors  will  be  practically 
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zero. 

If  matte  blue  and  red  objects  are  placed  next  to  each  other,  they  will  not  show  a  hue  shift  since  the 
body-to-body  reflection  is  insignificant.  Their  color  histograms  will  show  this  fact.  A  hue  shift  would  cause 
pixels  to  fall  outside  the  dichromatic  plane,  causing  the  formation  of  the  interreflection  cluster.  If  there  is  no 
body-to-body  reflection  and  no  hue  shift,  then  all  pixels  of  the  two  objects  will  fall  within  the  dichromatic 
plane.  If  it  is  assumed  that  the  objects  are  completely  matte,  even  those  points  on  the  objects  that  face  each 
other  will  be  located  on  the  body  reflection  clusters  in  the  histograms. 

For  the  case  of  body-to-body  reflection  between  a  red  object  and  a  yellow  one,  the  overlap  color  will  be 
red.  This  is  because  both  of  these  colors  reflect  a  lot  of  light  in  the  red  region  of  the  spectrum.  Although  the 
yellow  color  has  a  lot  of  reflectance  in  the  green  portion  of  the  spectrum,  the  red  has  practically  none  here. 
Neither  color  shows  any  appreciable  reflectance  in  the  blue  portion  of  the  spectrum. 

Consider  what  happens  when  the  red  body-to-body  interreflection  is  added  to  the  simple  body  reflection 
of  each  of  the  objects.  For  the  red  object,  there  actually  is  no  noticeable  hue  change  as  a  result  of  this  type 
of  interreflection.  This  is  because  both  the  body  and  the  body-to-body  components  are  red,  and  the  result 
of  adding  red  light  plus  some  more  red  light  merely  yields  red  light.  Pixels  that  are  influenced  by  this  color 
overlap  will  be  brighter  than  they  would  have  otherwise  been,  but  the  hue  will  be  the  same.  Figure  4.10 
shows  the  spectral  product  of  the  red  and  yellow  colors  and  also  shows  what  happens  when  this  overlap  color 
is  added  to  the  original  red  color.  The  resulting  color  is  essentially  just  a  scaled  up  version  of  the  original 
red  color. 

On  the  other  hand,  if  the  red  body-to-body  interreflection  is  added  to  the  simple  body  reflection  of  the 
yellow  object,  a  hue  change  will  be  noticed.  Figure  4.10  shows  that  when  the  red  overlap  color  is  added  to 
the  yellow  color,  the  resulting  spectral  reflectance  curve  has  a  significantly  different  shape.  Thus  pixels  on 
the  yellow  object  that  have  significant  amounts  of  body-to-body  reflectance  will  have  a  noticeably  different 
hue. 

The  conclusion  from  this  is  that  although  the  body-to-body  reflection  color  is  symmetrical,  the  influence 
on  object  appearance  is  not.  If  matte  red  and  yellow  objects  are  placed  next  to  each  other,  the  resulting 
histogram  will  show  an  interreflection  cluster  for  the  yellow  object,  corresponding  to  the  pixels  that  are 
hue-shifted.  But  the  histogram  for  the  red  object  will  have  no  interreflection  cluster.  Any  pixels  that  have 
body-to-body  reflection  will  have  the  same  red  hue  as  pixels  without  interreflection.  As  a  result  these  pixels 
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will  fall  on  the  body  reflection  cluster. 

The  interreflection  cluster  on  the  histogram  for  the  yellow  object  will  correspond  to  those  points  on  the 
surface  that  receive  increasing  amounts  of  the  body-to-body  reflection  color  of  red.  As  a  point  receives 
more  body-to-body  reflection,  it  will  move  further  from  the  yellow  body  reflection  cluster,  in  the  direction 
of  increasing  amounts  of  red.  As  described  in  [Hunt  and  Drew,  1991]  and  [Jang,  1991],  these  pixels  will 
fall  on  a  plane  defined  by  the  simple  body  reflection  color  (yellow)  and  the  overlap  color  (red).  Since  the 
overlap  color  is  essentially  the  same  as  the  simple  body  reflection  color  of  the  red  object,  the  pixels  will  fall 
on  the  plane  between  the  two  vectors  fitted  to  the  body  reflection  clusters. 

Finally,  consider  the  case  where  the  two  objects  of  interest  are  yellow  and  purple.  Figure  4.9  shows  that 
the  yellow  color  has  high  reflectance  in  the  red  and  green  regions  of  the  spectrum,  while  the  purple  color  has 
high  reflectance  in  the  blue  and  red  portions  of  the  spectrum.  This  means  that  the  color  overlap  will  again 
have  a  red  color.  However  this  time,  the  overlap  color  is  different  from  both  the  object  colors. 

For  the  case  of  interreflection  between  matte  yellow  and  purple  objects,  both  objects  will  have  a  body- 
to-body  reflection  component  of  red.  Thus  both  objects  will  have  a  noticeable  hue  shift.  This  will  be 
manifested  in  the  histograms  as  interreflection  clusters  that  deviate  from  the  body  reflection  clusters.  Both 
clusters  will  be  in  the  direction  of  increasing  amounts  of  red.  For  the  yellow  object,  the  pixels  experiencing 
interreflection  will  fall  on  the  plane  defined  by  the  vector  fitted  to  the  body  reflection  cluster  and  the  vector 
that  points  in  the  direction  of  the  red  overlap  color.  Similarly  for  the  purple  object,  the  interreflection  pixels 
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will  fall  on  the  plane  defined  by  the  vectors  for  the  purple  bods  color  and  the  red  os  eriap  color 

Taken  together,  the  interrefiection  pixels  front  both  surfaces  lorni  pkuies  that  intersect  at  the  overlap 
color.  This  IS  the  situation  desenbed  in  iFunt  and  Drew.  1991].  They  use  this  fact  to  autoniatjculh  detect  the 
overlap  color  for  matte  objects,  it  can  then  be  subtracted  from  pixels  to  lonn  an  image  u  uh  the  interrefiection 
removed.  However  this  techniques  only  vvorks  if  the  overlap  color  is  different  from  hsuh  body  refiection 
colors,  and  it  also  assumes  completely  matte  objects. 


4.3.3  Body  to  Surface  Reflection 

Many  objects  in  the  real  world  are  not  matte.  Shiny  plastic  oi  painted  objects  can  mimic  a  mirror,  giving  a 
very  clear  reflection  of  nearby  objects.  This  effect  is  a  result  of  the  body  -lo-surface  component  of  reflection 
It  is  especially  noticeable  for  shiny  objects  that  are  oark,  where  the  body  reflection  is  low  and  does  not 
distract  the  eye. 

Surface  reflection  is  ahvays  present  in  inhomogeneous  objects.  .4ny  time  light  encounters  an  interface 
with  a  change  m  the  index  of  refraction,  some  of  the  light  will  be  immediately  reflected  back  at  the  interface. 
Thus  even  the  matte  objects  that  have  been  modeled  as  Lambertian  will  experience  surface  reflection.  These 
surfaces  are  optically  rough  enough  that  the  surface  reflection  is  scattered  in  every-  direction,  making  it 
indistinguishable  from  body  reflection.  This  explains  why  a  black  shiny  object  looks  darker  than  a  black 
matte  one.  With  a  matte  black  object,  some  of  the  surface  reflection  is  scattered  in  the  direction  of  the  eye. 
making  the  object  look  lighter.  A  shiny  black  object  reflects  the  surface  reflection  in  a  paniculai  direction; 
if  the  surface  is  viewed  from  some  other  direction,  that  surface  reflection  is  not  seen,  making  the  object  look 
darker. 

If  an  object  is  shiny,  it  may  reflect  some  of  the  color  of  a  neighboring  object  directly  into  the  camera. 
Consider  a  red  object  again,  but  this  time  a  shiny  one.  fr  a  yellow  object  is  nearby,  some  of  *ae  yellow 
body  reflection  from  it  may  be  reflected  at  the  surface  of  the  red  object,  causing  the  appearance  of .)  yellow- 
reflected  image  on  the  red  surface.  This  will  cause  a  noticeable  hue  shift  on  the  object,  and  hence  affect  the 
histogram. 

The  yellow  bcdy-to-surface  component  of  interreflected  pixels  is  added  to  the  other  reflection  components 
at  that  point.  It  will  be  added  to  the  red  color  from  the  simple  body -reflect  ion  and  also  to  the  red  color  of 
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the  body-io-body  reflection.  In  the  histogram,  these  pixels  will  form  an  inteneflectiori  cluster  that  deviates 
from  the  body  reflection  cluster,  in  the  direction  of  increasing  amounts  ot  yellow’. 

If  the  surface  of  the  red  object  is  optically  rough,  it  will  still  have  a  body-to-surface  component  from 
the  yellow'  object.  However  it  will  scatter  that  reflection  wndely  so  that  most  of  it  misses  the  camera.  In  this 
case  there  will  be  no  noticeable  influence  from  the  yellow  object,  and  hence  no  inteneflection  cluster  m  the 
histogram.  This  means  that  the  deviation  of  the  interreflcction  cluster  from  the  dichromatic  plane  depends 
not  only  upon  the  colors  of  the  two  objects,  but  also  their  degree  of  roughness. 

If  both  the  body-to-body  reflection  and  the  body-to-surface  reflection  are  high,  the  resulting  histogram 
depends  upon  the  body  color  of  the  other  object  and  the  overlap  color  of  the  two  objects.  In  the  case  of  the 
shiny  yellow  object  next  to  the  red  object,  these  are  the  same,  since  the  body-to-surface  color  and  overlap 
color  are  both  red.  Both  components  of  interreflection  will  cause  the  interreflection  cluster  to  lie  on  the  plane 
defined  by  the  yellow  and  red  vectors. 

However,  for  some  objects  these  two  components  will  have  different  colors.  This  is  the  situation  when 
a  shiny  yellow  object  is  next  to  a  purple  one.  The  body-to-body  color  will  have  a  red  hue  whereas  the 
body-to-surface  color  will  have  a  purple  hue.  In  this  case  the  mterreflection  cluster  may  not  lie  neatly  m  a 
plane,  but  fall  between  the  plane  defined  by  the  yellow  and  red  vectors  and  the  plane  defined  by  the  yellow 
and  purple  vectors. 


4  J.4  Analysis  of  the  Interreflection  Cluster 

After  describing  the  influence  of  each  of  the  components  of  interreflection.  it  is  now  obvious  that  the 
interreflection  cluster  in  the  histogram  is  affected  by  several  things.  If  it  is  known  from  scene  geometry  that 
interreflection  may  occur,  and  the  body  colors  and  roughness  values  of  the  objects  mvolved  are  also  known, 
one  could  reason  forward  and  say  how’  the  colors  will  be  distributed  in  the  histogram.  This  is  essentially  a 
graphics  problem,  where  the  objects  are  modeled  first,  and  then  the  image  is  created.  In  vision  problems 
researchers  start  with  the  image  and  want  to  generate  a  model  of  the  objects. 

However,  after  examining  the  interreflection  cluster  of  a  histogram,  it  is  possible  to  make  some  predictions 
about  the  object.  Section  4.3.2  showed  that  the  interreflection  cluster  will  be  abt  mt  if  the  objects  are  matte 
and  there  is  no  color  overlap  between  the  colors,  or  if  the  color  overlap  has  the  same  color  as  the  body  color. 


43.  THE  INTERREFLECTION  CLUSTER  73 

This  thesis  has  already  shown  how  to  estimate  object  roughness  from  other  histograxn  features.  If  the  object 
has  been  determined  to  be  optically  rough,  then  the  absence  of  an  interretlection  cluster  puts  constraints 
upon  what  color  objects  could  be  close  by. 

If  on  the  other  hand,  analyzing  other  features  in  the  histogram  has  indicated  that  the  object  is  very 
smooth,  then  it  is  likely  to  have  body-io-surface  reflection.  The  plane  formed  by  the  inietrefleciion  cluster 
will  indicate  directly  the  color  of  the  other  object. 

-Alternatively  if  the  colors  of  both  objects  are  known,  and  the  mterreflection  cluster  has  been  identified 
from  the  image,  it  might  be  possible  to  break  down  the  image  mto  interreflection  components.  This  analysis 
of  the  histogram  would  enable  one  to  remove  not  only  the  influence  of  body-to-body  reflection  as  is  shown  m 
[Flint  and  Drew,  1991],  hut  also  the  other  components  that  are  present  when  surface  reflection  is  considered. 

The  end  result  would  be  additional  “mtrinsic  images."  Shafer  origmally  proposed  breaking  down  an 
image  with  highlights  into  an  mtnnsic  image  of  body  reflection  and  an  intrinsic  image  of  surface  reflection 
[Shafer,  1984],  Klinker  showed  that  these  images  may  be  automatically  generated  from  the  onginal  image, 
showing  what  objects  would  look  like  if  they  were  completely  matte  (body  reflection  only)  and  how  they 
would  appear  if  made  of  a  shmy  black  material  (surface  reflection  only)  [Klinker,  19881.  However,  the 
method  assumed  no  interreflection.  An  analysis  of  the  components  of  interreflection  could  be  used  to  show 
how  an  object  with  highlights  will  appear  if  nearby  objects  are  removed  from  the  scene.  It  might  also  be 
used  to  show  how'  the  interreflection  would  appear  if  the  objects  were  completely  matte,  or  how  it  w’ould 
look  between  objects  that  only  exhibit  surface  reflectance.  Automatic  analysis  of  images  with  mterreflection 
is  a  subject  for  future  work. 
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Chapter  5 


Analysis  of  the  Color  Histogram  for  an 
Ideal  Sensor 


Chapters  3  and  4  described  the  relationship  between  the  shape  of  the  color  histogram  and  various  interesting 
scene  properties.  Understanding  the  relationship  is  the  first  step  in  analyzing  color  histograms.  The  next  step 
is  figuring  out  how  to  actually  exploit  the  histogram  to  recover  quantitative  measures  of  the  scene  properties. 

This  chapter  will  assume  an  ideal  sensor  that  has  highly  desirable  properties.  This  kind  of  camera, 
which  is  unobtainable  in  practice,  shows  the  limit  of  what  could  be  obtained  under  the  best  possible  imaging 
conditions.  Chapter  6  will  show  how  the  requirements  may  be  relaxed  for  the  case  of  a  more  realistic  camera. 


5.1  Properties  of  an  Ideal  Sensor 

For  the  purposes  of  this  thesis,  an  ideal  sensor  is  one  that  obeys  the  imaging  equation 

Px  =  i  j  L(X,e.,er,ep)SA^)dx  (s.i) 

where is  the  pixel  measurement  in  each  band  and  x  is  one  of  red,  green,  or  blue.  S;t(A)  is  the  responsivity 
of  the  sensor  in  each  of  the  bands;  the  integration  is  performed  over  those  wavelengths  where  Sx  is  greater 
than  zero. 

Tyrpically  the  non-zero  wavelengths  are  assumed  to  be  those  wavelengths  of  light  that  are  visible  to 
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humans,  but  that  need  not  be  the  case  here.  Many  of  the  CCD  cameras  used  in  image  understandmg  are 
very  sensitive  to  infrared  wavelengths  and  may  result  in  images  that  look  somewhat  surprising  to  humans 
[Novak  ei  al.,  1990].  How'ever  that  does  not  affect  the  dichromatic  reflection  model,  as  long  as  the  camera 
is  able  to  sense  wavelengths  where  there  is  a  difference  in  the  amount  of  light  caused  by  body  reflection  as 
opposed  to  surface  reflection. 

Equation  5.1  implies  that  the  camera  behaves  linearly  since 

j  nL{X.e,.er.ep)Sx(>^)dX  =  n  j  L{X.d,.e,,ep)SAMdX  (5.2i 

for  all  constants  «.  This  means  that  if  the  light  intensity'  or  surface  reflectance  is  mcreased,  the  pixel  value 
will  be  proponionally  larger.  In  practice,  many  cameras  are  reasonably  linear  in  their  response.  Even  w'hen 
they  are  not,  a  linear  correction  is  easy  to  apply  once  the  non-linearity  has  been  measured  [Novak  et  al., 
1990], 

Equation  5.2  also  implies  that  the  camera  has  an  infinite  dynamic  range  since  n  may  equal  10,  100  or 
even  a  million.  Obviously  real  cameras  do  not  have  this  property;  they  have  some  maximum  value  that  can 
be  measured.  After  this  maximum  has  been  reached,  increasing  the  brighmess  of  a  pomt  will  not  cause  an 
increase  in  the  corresponding  pixel  value.  The  result  is  that  the  pixel  value  is  “clipi>ed'"  at  the  maximum 
value  of  the  camera.  In  the  ideal  model,  clipping  does  not  occur. 

When  taking  real  pictures,  researchers  can  usually  adjust  the  aperture,  light  level,  or  exposure  time  so  that 
the  brightest  point  in  the  image  is  within  the  dynamic  range  of  the  camera.  However  this  presents  problems 
in  scenes  that  contain  highlights.  Highlights  are  much,  much  brighter  than  surrounding  non-highlight  areas, 
so  that  if  the  camera  is  adjusted  to  measure  highlights  properly,  the  non-highlight  areas  will  be  so  dim  as  to 
be  lost  in  the  camera  noise. 

Since  it  is  precisely  pictures  w'lih  highlights  that  this  thesis  is  concerned  with,  the  assumption  of  unlimited 
dynamic  range  is  much  less  realistic  than  the  linearity  assumption.  This  assumption  will  be  removed  in  the 
next  chapter. 

A  final  assumption  made  about  the  ideal  sensor  is  that  it  does  not  suffer  from  any  noise  problems. 
Shot  noise,  dark  current  noise,  and  pre-amplifier  noise  are  three  sources  of  noise  that  occur  in  real  CCD 
cameras  [Photometries,  1989],  [Healey  and  Kondepudy,  1991].  Equation  5.1  does  not  take  these  effects  into 
consideration.  In  the  next  chapter  these  effects  will  be  modeled  and  taken  into  consideration  in  the  analysis 
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of  histograms. 


5.2  Knowns  and  Unknowns 

The  known  image  parameters  which  can  be  measured  from  the  color  histogram  of  a  single  object  are  the 
body  reflection  cluster's  length  and  direction;  the  highlight  cluster’s  length,  width,  and  direction;  and  the 
intersection  point  of  the  two  clusters.  This  gives  four  scalar  values  and  two  vector  quantities.  They  will  be 
referred  to  by  the  following  variables: 

•  I  •  length  of  highlight  cluster 

•  w  -  widdi  of  highlight  cluster 

•  i  •  inters ‘ction  of  two  clusters 

•  b  -  leng»  j  of  body  reflection  cluster 

•  dj  -  dire,  fion  of  highlight  cluster 

•  db-  dire.aon  of  body  reflection  cluster 

The  unknown  scene  parameters  which  are  to  be  recovered  from  the  histogram  can  also  be  divided  into 
scalar  values ..  id  vector  quantities.  The  scalar  values  are  the  surface  roughness,  the  phase  angle  between  the 
light  source  di  ection  and  camera  direction,  the  light  intensity,  and  the  surface  albedo.  The  vector  quantities 
are  the  chrom  deity  of  the  illumination  and  that  of  the  object.  These  variables  are: 

•  cr  -  optical  roughness 

•  dp  -  phase  angle 

•  By  -  illumination  intensity 

•  Bo  -  object  albedo 

•  cl  -  chromaticity  of  light  source 

•  cl  -  chromaticity  of  object  under  “white”  light 
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For  convenience,  the  illumination  and  object  reflectance  have  been  separated  into  intensity  components 
and  chromatic  components.  As  described  in  section  3.1  the  object's  chromaticit\'  cl  may  be  recovered  in 
a  straightfori^’ard  way  from  the  direction  of  the  body  reflection  cluster  dt,  and  the  color  of  the  light  source 
cj.  The  red  component  of  the  body  reflection  cluster  is  divided  by  the  red  component  of  the  light  source 
color,  the  green  component  divided  by  the  green  component,  and  the  blue  component  divided  by  the  blue 
component. 

=  4  cf] 

The  result  is  then  normalized  to  length  1.  This  recovers  only  two  parameters  of  the  object's  reflectance  {a 
third  parameter  is  albedo),  but  this  is  a  useful  approximation  in  most  cases,  since  most  objects  have  smoothly 
varying  reflectance  fimctions. 

The  albedo  of  the  object  Bp  is  also  recovered  in  a  straightforward  manner  from  the  length  of  the  body- 
reflection  cluster  b  and  the  illumination  intensity' Sj.  From  equation  2.1  the  body  reflection  magnitude  is 

mb  =  -yBbCosiO,) 

where  Bb  is  the  object’s  apparent  brighmess.  The  body  cluster  length  b  is  equal  to  the  maximum  value  of 
mb,  which  will  occur  when  cosiOi)  =  1.  Therefore  b  =  yB*.  Since  the  apparent  brightness  of  the  object  is 
the  product  of  the  object  albedo  Bo  and  the  illumination  intensity  Bj,  the  albedo  is 

Bo  =  Bb!Bi  -  b/(,-yBi) 

For  the  remaining  unknowns  the  situation  is  not  so  simple.  In  chapter  3  it  was  shown  that  the  length  / 
is  related  to  surface  roughness  and  illumination  intensity,  but  is  also  dependent  upon  imaging  geometry  and 
even  the  illumination  color.  The  remaining  knowns 

•  /  -  length  of  highlight  cluster 

•  w  -  width  of  highlight  cluster 

•  i  -  intersection  of  two  clusters 

•  dj  -  direction  of  highlight  cluster 


and  unknowns 


5J.  ANALYTIC  SOLUTION 


79 


•  £T  -  optical  roughness 

•  Op  -  phase  angle 

•  B,  -  illumination  intensity 

•  cl  -  chromaticity  of  light  source 

will  be  examined  in  detail  in  the  next  few  sections. 


5.3  Analytic  Solution 


Equations  2.1  and  2.3  describe  the  amounts  of  body  and  surface  reflection  mi,  and  as  a  fimction  of  imaging 
geometry  and  light  intensity  Bs-  Equation  2.3  also  shows  how  the  amount  of  surface  reflection  at  a  given 
point  varies  with  the  roughness  of  the  surface  a.  The  equations  again  are: 


mb 


‘)Bb  cos{0i) 

„  f  e,)  a 


exp 


As  mentioned  in  section  3.2.2,  the  Fresnel  coefficient  will  be  assumed  to  be  1.5.  The  Fresnel  coefficient  is 
only  weakly  dependent  upon  wavelength  and  incidence  angle,  and  is  very  close  to  1.5  for  nearly  all  paints 
and  plastics.  The  constant  a  includes  the  facet  size  and  some  constants  of  integration.  The  facets  of  the 
surfaces  in  question  will  be  assumed  to  be  of  uniform  size. 


Unfortunately  it  is  not  possible  to  directly  solve  for  the  scene  parameters  from  the  histogram  measure¬ 
ments.  The  length  of  the  highlight  cluster  indicates  the  maximum  amount  of  surface  reflection  seen  anywhere 
on  the  object.  For  given  values  of  cr.  Op,  and  the  length  /  may  be  calculated 


/ 


MAX 


B  PG{0„er,0p)Q 

^  cr  cos(0r) 


exp 


over  all  values  of  0,,0r,  0^ 


(5.3) 


If  it  were  not  for  the  presence  of  G  in  the  numerator  and  cos(6;.)  in  the  denominator,  the  length  would  be 
easily  derived.  If  they  disappeared  from  the  equation,  the  maximum  value  of  the  surface  reflectance  would 
occur  at  the  perfect  specular  angle  when  6^  is  zero. 


In  order  to  derive  the  roughness  from  the  length  measurement  with  any  accuracy,  it  is  necessary  to 
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consider  the  influence  of  G  and  1  /  cos(0r).  Recall  from  equation  2.4  that  the  formula  for  C  is  given  as 


G 


MIN 


ll  2 cos(^i) cos(^r)  2cos(^^)cos(^,) 
[  ’  cos(6p)  '  cos{6p) 


(5.4) 


However  G  is  not  the  sole  source  of  difficulty  for  a  direct  solution.  It  is  significant  that  the  roughness 
parameter  cr  occurs  both  in  the  denominator  and  in  the  exponential  of  equation  5.3.  Ev  en  if  all  terms  besides 
<T  were  assumed  to  be  constant  and  equal  to  1,  the  equation  would  reduce  to 

/  =  -exp(--^) 
a  cr- 

which  has  no  analytic  solution  for  cr.  Clearly  a  direct  solution  of  the  complete  surface  reflection  equation 
caimot  be  obtained. 


5.4  Ideal  vs.  Measured 

Although  equation  5.3  has  no  analytic  solution,  it  might  be  possible  to  solve  it  iteratively,  through  some  sort 
of  search  (for  example  by  gradient  descent).  However  that  assumes  the  length  /  can  be  accurately  measured 
from  the  histogram.  In  section  3.2.2,  the  length  is  measured  from  the  tip  of  the  highlight  cluster  to  its 
base  along  the  direction  of  the  highlight  color.  Unfortunately  the  highlight  color  is  not  typically  known 
in  advance.  The  highlight  color  may  be  estimated  by  fitting  a  vector  to  the  pixels  that  form  the  highlight 
cluster,  but  as  described  in  section  3.2.1,  that  estimate  may  be  skewed. 

This  is  shown  in  figure  5.1,  which  is  the  histogram  of  a  simulated  rough  object  illuminated  with  white 
light.  The  dotted  “measured”  line  shows  the  direction  calculated  for  the  best  fit  vector  to  the  highlight 
cluster.  In  this  case  the  best  fit  line  to  the  cluster  will  not  pass  through  the  brightest  point  in  the  highlight 
cluster.  The  position  of  the  dotted  line  in  figure  5.1  shows  the  projection  of  the  brightest  highlight  pixel  onto 
the  body  reflection  vector  along  the  best  fit  vector,  since  the  the  length  is  calculated  from  the  brightest  pixel. 
The  “ideal”  line  indicates  the  direction  of  the  actual  illumination  color.  The  skewing  causes  the  measured 
length  to  be  longer  than  it  would  have  been  if  the  correct  illumination  color  had  been  known. 

This  contrasts  the  concept  of  “ideal”  length  that  one  would  like  to  obtain  from  the  histogram,  with  the 
“measured”  length  that  can  actually  be  recovered  without  a  priori  knowledge  about  the  illumination  color. 
All  the  graphs  presented  in  chapter  3  that  relate  highlight  cluster  length  to  roughness  or  imaging  geometry 
used  the  ideal  length,  obtained  by  knowing  the  illumination  color. 
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Figure  5.1;  Skewed  length  measurement 

Figure  5.2  shows  how  both  the  ideal  length  and  measured  length  vary  with  roughness  for  the  case  where 
the  camera  and  light  source  are  separated  by  20  degrees.  For  smooth  surfaces  both  values  show  an  inverse 
relationship  between  length  and  roughness.  As  the  roughness  increases,  the  deviation  between  the  two 
curves  grows;  eventually  the  measured  length  begins  increasing  as  the  amoimt  of  roughness  is  increased. 
If  only  the  measured  length  value  is  available,  it  would  be  difficult  to  tell  a  very  smooth  surface  from  a  very 
rough  one. 

The  variation  of  the  “ideal”  length  with  roughness  and  imaging  geometry  is  shown  in  the  surface 
presented  in  figure  5.3.  The  length  is  shown  as  the  height  of  the  surface,  while  incremental  changes  in 
roughness  are  shown  along  the  x-axis  and  changes  in  phase  angle  are  shown  along  the  y-axis. 

Figure  5.4  shows  the  variation  in  the  “measured”  length  with  step  changes  in  siuface  roughness  and 
phase  angle.  It  looks  somewhat  different  than  the  graph  shown  in  figure  5.3. 

A  similar  situation  occurs  with  the  measurement  of  width  and  intersection  point.  When  the  highlight  is 
skewed,  the  values  measured  from  the  histogram  will  be  different  from  the  ideal  values  shown  in  the  graphs 
in  chapter  3.  The  intersection  point  is  measured  by  taking  the  brightest  pixel  in  the  highlight  cluster  and 
projecting  it  onto  the  body  reflection  vector.  This  projection  is  done  along  the  vector  that  was  assumed  to  be 
the  highlight  cluster.  In  the  absence  of  a  priori  information  about  the  illumination  color,  the  best-fit  vector 
to  the  cluster  will  have  to  be  used. 

The  point  of  intersection  with  the  body  reflection  cluster  is  supposed  to  represent  the  amount  of  body 
reflection  at  the  maximum  highlight  point.  However,  if  the  highlight  cluster  is  skewed  towards  the  body 
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Figure  5.3:  Variation  in  ideal  length  with  changes  in  roughness  and  phase  angle 
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Figure  5.4:  Variation  in  measured  length  with  changes  in  roughness  and  phase  angle 


reflection  cluster,  the  highlight  pixel  will  be  projected  too  far  down,  yielding  an  amount  of  body  reflection 
that  is  too  small.  In  some  cases,  the  amoimt  of  predicted  body  reflection  will  be  negative.  This  is  shown  in 
figure  5.5  where  a  histogram  is  shown  for  a  simulated  rough  surface  with  a  =  16®  and  9p  =  90®.  The  “ideal” 
line  shows  the  direction  of  the  actual  illumination  color  and  where  it  intersects  the  body  reflection  cluster 
when  projected  from  the  brightest  point  in  the  highlight.  The  “measured”  line  shows  the  direction  of  the 
vector  fitted  to  the  highlight  cluster,  as  well  as  the  projection  from  the  brightest  point  along  that  vector. 

Chapter  3  argued  that  intersection  represents  the  amount  of  body  reflection  seen  at  the  highlight  peak  as 
a  fraction  of  the  maximum  amount  of  body  reflection  seen  anywhere  on  the  object.  Logically  speaking,  it 
must  fall  between  0  and  1,  since  the  amount  of  body  reflection  will  never  be  negative.  However  the  method 
used  for  measuring  the  amoimt  of  body  reflection  relies  on  the  vector  fitted  to  the  highlight  cluster.  If  this 
vector  deviates  from  the  correct  highlight  color,  the  intersection  measurement  will  be  off,  and  may  even 
yield  a  negative  value. 


Figures  5.6  and  5.7  show  how  the  ideal  and  measured  intersection  values  vary  with  changes  in  the  surface 
roughness  and  imaging  geometry.  Figure  5.7  shows  that  there  is  a  range  of  roughness  values  and  phase 
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measured 

ideal 


Figure  5.5:  Histogram  with  a  negative  intersection 


angles  for  which  the  measured  intersection  will  be  negative. 

By  the  same  token,  the  width  measurement  is  also  affected  by  skewing  of  the  highlight  cluster.  The 
width  is  supposed  to  represent  the  range  of  body  reflectance  values  which  show  a  highlight  compared  to  the 
range  of  body  reflectance  values  seen  anywhere  on  the  object.  So  it  too  should  be  a  fraction  between  0  and 
1.  However  if  some  parts  of  the  highlight  are  calculated  as  having  a  negative  amount  of  body  reflectance, 
while  others  are  calculated  as  having  the  maximum  amount,  the  total  range  will  be  greater  than  the  range 
of  body  reflectance  values,  giving  a  width  measiuement  greater  than  1.  An  example  of  this  is  shown  in 
figure  5.8  which  shows  the  same  histogram  from  figure  5.5.  Projecting  all  highlight  pixels  down  along  the 
measured  highlight  vector  gives  a  width  measurement  greater  than  1 . 

Figures  5.9  and  5.10  show  how  the  idea)  and  measured  width  values  vary  with  changes  in  the  surface 
roughness  and  imaging  geometry.  Figure  5.10  shows  the  range  of  roughness  values  and  phase  angles  for 
which  the  measured  width  will  exceed  1 . 

In  the  absence  of  a  priori  information  about  the  illumination  color,  a  system  can  only  measure  what  is 
available  in  the  histogram.  How  can  the  ideal  values  be  derived  from  the  ones  that  are  measured?  Once  that 
is  done  how  can  the  image  parameters  of  interested  be  recovered? 
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Figure  5.10;  Vanation  of  measured  width  values 


5.5  Approximate  Solution 

The  approach  taken  by  this  thesis  is  to  recover  scene  parameters  by  an  approximate  method,  directly  from  the 
initial  histogram  measurements.  Therefore,  it  is  not  necessary’  to  recover  “ideal”  histogram  values  from  the 
“measured”  ones.  The  ideal  values  of  the  histogram  are  a  useful  abstraction  smce  then  relationship  to  scene 
parameters  is  easy  to  explam.  However,  they  cannot  be  obtained  from  the  histogram  without  knowledge  of 
the  illumination  c«^iOr. 

The  surfaces  shown  in  figures  5.4, 5.7.  and  5.10  show  the  variation  in  measured  length,  mtersection,  and 
width  as  surface  roughness  and  phase  angle  are  varied.  A  given  value  of  length  descnbes  a  contour  within 
the  space  of  roughness  and  phase  angles.  The  values  for  these  scene  parameters  must  lie  somewhere  on  that 
contour.  The  additional  measurements  of  width  and  intersection  also  have  contours  in  the  roughness-phase 
angle  space.  For  a  given  histogram  with  its  particular  length,  width,  and  mtersection  measurements,  the 
contours  may  have  a  unique  intersection  in  the  space  defined  by  roughness  and  phase  angle. 

The  preceding  discussion  left  out  ‘he  additional  scene  parameter  of  light  intensity.  If  that  is  considered 
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in  addition  to  surface  roughness  and  phase  angle,  these  three  scene  parameters  form  a  three  dimensional 
space.  A  length  measurement  would  then  describe  a  two  dimensional  surl'ace  within  that  space,  showing  the 
possible  roughness,  phase  angle,  and  light  mtensiU’  values  that  could  give  nse  to  a  histogram  with  that  length 
highlight  cluster.  The  intersection  and  width  measurements  would  also  descnbc  a  set  of  surfaces  w  uhm  that 
space.  The  hope  is  that  the.se  three  surfaces  intersect  at  a  single  point  in  that  space,  making  it  possible  to 
recover  a  unique  value  for  surface  rougliness,  phase  angle,  and  illumination  intensity.  So  obvious  questions 
are:  how'  can  these  contours  of  equal-length,  equal-width,  and  equal-intersection  be  obtained;  and  do  these 
contours  intersect  to  give  unique  solutions’’ 

Section  5.3  pointed  out  that  there  is  no  analytic  solution  to  generate  the  contours.  How'cver.  the  graph  m 
figure  5.4  shows  how  the  highlight  cluster  length  varies  with  roughness  and  phase  angle  at  discrete  pomts. 
These  values  came  from  simulatmg  an  object  with  those  parameters  and  then  measurmg  the  length  of  the 
highlight  cluster  in  the  the  resuhmg  color  histogram.  Obviously  a  detailed  lookup  table  can  be  created  by 
simulatmg  a  large  range  of  roughness  values,  phase  angles,  and  illumination  mtensities.  and  then  storing  the 
corresponding  histogram  measurements  of  length,  width,  and  intersection  in  the  table.  Given  a  histogram 
associated  with  unknown  scene  parameters,  the  lookup  table  is  searched  to  find  the  corresponding  scene 
parameters. 

The  more  interesting  quesuon  is  whether  a  unique  solution  exists  for  a  given  triple  (/.  u .  i).  If  some  tnple 
had  more  than  one  solution,  that  would  mean  that  different  combinations  cf  scene  parameters  could  give  rise 
to  identical  histogram  measurements  It  would  also  mean  that  a  search  through  the  contours  in  parameter 
space  could  not  be  guaranteed  to  converge. 

To  visualize  the  distribution  of  possible  (/,  w.  i)  triples,  figure  5.11  show’s  the  space  of  all  possible  values 
for  length,  width  and  intersection.  The  x-axis  encodes  the  range  of  highlight  cluster  lengths,  me  y-axis 
shows  the  range  of  widths,  and  the  z-axis  show's  intersections.  The  surface  wnhm  it  is  defined  b>  lines  of 
equal  roughness  in  one  direction  and  lines  of  equal  phase  angle  m  anothei  direction.  (In  this  figure,  the 
illumination  intensity  is  constant.)  This  surface  shows  which  combmations  of  length-w  idth-mtersectionare 
possible  for  histograms  within  the  simulated  range.  Points  in  the  space  not  falling  on  the  surface  correspond 
to  histograms  that  do  not  make  sense  and  thus  cannot  arise  from  real  images. 

Although  the  surface  curves  around  in  the  L-W-I  space,  it  does  not  intersect  itself  anywhere.  This 
means  that  any  triple  of  length-width-intersection  that  falls  on  the  surface  is  associated  with  a  unique  set  of 
surface  roughness  and  phase  angle  values.  The  only  remaining  problem  is  to  determine  from  these  histogram 
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Figure  5.1 1;  Surface  of  roughness  and  phase  angles  within  L-W-I  space 
measurements  where  on  the  surface  they  will  lie. 

The  surface  in  figure  5.11  shows  only  a  range  of  roughness  values  and  phase  angles.  If  the  range  of 
illumination  intensities  were  also  shown,  that  would  give  rise  to  a  three  dimensiontil  curved  solid  within  the 
L-W-I  space. 

5.6  Generating  Lookup  Tables 

A  large  range  of  roughness  values,  phase  angles,  and  illumination  intensities  are  used  to  create  the  lookup 
tables.  The  ranges  are  shown  in  the  table  in  figure  5.12.  A  roughness  value  is  the  standard  deviation  of 
facet  angles  with  respect  to  the  global  surface  normal,  in  degrees.  The  phase  angle  is  the  angle  between  the 
camera  and  light  source  with  respect  to  the  object,  also  in  degrees.  The  light  intensity  is  a  percentage  of  a 
hypothetical  light’s  maximum  output. 

For  each  set  of  roughness,  phase  angle,  and  light  intensity  values,  a  simulated  object  is  generated,  and  the 
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Minimum 

Maximum 

Increment 

Total  Used 

Roughness 

1* 

15= 

8 

Phase  Angle 

0= 

90- 

10^ 

10 

Intensity 

50% 

100% 

10% 

6 

Overall 

480 

Figure  5.12:  Range  of  scene  parameters  used  to  generate  lookup  table 


Farthest  Pom: 


Brightest  Point 


Figure  5.13;  Finding  the  body  cluster 


histogram  from  the  resulting  image  is  measured  to  obtain  the  dimensions  to  be  entered  into  the  lookup  table. 
The  simulation  employs  the  noise  free  camera  equation  5.1,  and  the  histogram  is  automatically  separated 
into  body  reflection  and  highlight  clusters.  The  technique  is  sunilar  to  that  described  by  Klinkcr  [Klmkcr. 
1988], 

First,  a  line  is  drawn  from  the  black  point  of  the  color  cube  to  the  brightest  point  in  the  histogram.  Then 
the  point  in  the  histogram  farthest  from  that  line  is  found.  A  second  line  is  drawn  from  the  black  pomt  to 
this  new  point.  One  of  these  two  lines  defines  the  body  reflection  cluster.  Most  often  it  is  the  second  Ime.  as 
shown  in  the  histogram  on  the  left  in  figure  5.13.  However  it  could  be  the  case  that  the  bnghtest  point  of  the 
histogram  would  be  on  the  body  reflection  cluster,  if  the  highlight  were  very  dim.  In  this  case  the  first  line 
is  the  correct  choice  for  defining  the  body  reflection  cluster.  This  situation  is  show-Ti  m  the  histogram  on  the 
right  in  figure  5.13. 

The  line  for  the  body  reflection  cluster  is  chosen  according  to  the  “saturation  rule”.  Surface  reflection  is 
assumed  to  have  the  color  of  the  illumination.  Body  reflection  is  assumed  to  be  the  product  of  the  illumination 
color  and  the  body  reflection  color.  Therefore  the  body  reflection  color  must  be  narrower-band  than  the 
highlight  color.  For  example,  if  the  illumination  does  not  have  any  blue  in  it,  than  the  body  reflection  cannot 
either,  even  if  the  object  has  a  color  that  we  would  call  blue.  If  the  illumination  does  have  blue  in  it.  the 
body  reflection  can  contain  as  much  blue  as  it  does  (if  the  object  reflects  100%  of  the  light  in  the  blue  range) 
or  less.  Therefore  the  line  that  represents  a  more  saturated  color  is  chosen  as  the  body  reflection  line. 
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It  is  possible  that  the  two  lines  are  equally  saturated  by  the  conventional  saturation  calculation  For  a 
normalized  color  [r,^.  fr],  the  saturation  S  is  defined  as 

S  =  1  -  5MIN[r.g,b} 

As  an  example,  a  yellow  light  and  a  red  object  both  have  a  saturation  of  1 ,  since  the  blue  component  is  zero. 
In  the  case  where  the  saturations  are  equal,  the  ’'saturation  rule”  is  extended  to  consider  only  the  remaining 
non-zero  components. 

S  =  1  -  2MIN(r.g) 

Since  red  is  more  saturated  than  yellow  by  this  definition,  the  red  line  would  be  chosen  as  the  body  reflection 
cluster.  The  vector  that  defines  the  chosen  line  becomes  db. 

Once  the  body  reflection  line  has  been  chosen,  any  pixels  that  are  more  than  the  noise  threshold  away 
from  the  body  reflection  line  are  assumed  to  be  in  the  highlight  cluster.  The  noise  threshold  may  be  chosen 
arbitrarily  for  the  noise-free  case,  but  some  threshold  must  be  chosen,  for  the  highlight  component  never 
quite  goes  to  zero  with  equation  2.3.  In  the  more  realistic  case  where  there  is  camera  noise,  as  will  be 
discussed  in  chapter  6,  the  threshold  will  be  based  on  the  level  of  that  noise.  Eigenvector  analysis  is  done 
on  all  pixels  classified  as  highlight  to  determine  the  principle  vector  of  the  cluster.  This  vector  is  d,. 

Because  the  vector  is  determined  by  the  best  fit  to  all  highlight  pixels,  it  is  not  completely  independent 
of  the  object  shape.  If  an  object  has  many  surface  normals  corresponding  to  pixels  on  one  side  of  the 
highlight  cluster,  and  only  a  few  corresponding  to  the  other,  that  may  bias  the  direction  of  the  vector.  This 
thesis  assumes  that  such  a  systematic  effect  rarely  occurs,  and  in  any  case  is  unlikely  to  cause  a  significant 
deviation  in  cluster  direction.  If  this  is  not  the  case,  an  alternate  method  of  determining  vector  direction, 
such  as  fitting  to  the  convex  hull  of  the  highlight  cluster,  could  be  used.  The  convex  hull  of  the  highlight 
cluster  depends  only  upon  a  broad  distribution  of  surface  normals;  this  is  also  true  of  the  length,  width,  and 
intersection  measurements. 

Once  the  direction  of  the  highlight  cluster  has  been  measured,  the  vector  ds  is  used  to  project  all  highlight 
pixels  onto  the  body  reflection  vector  d* .  These  projections  determine  the  relative  contributions  of  the  vectors 
in  each  pixel.  Each  pixel  p  in  the  histogram  can  then  be  defined  as 

p  =  mJs  +  mbdb 

This  is  essentially  the  dichromatic  equation,  but  the  highlight  cluster  direction  ds  may  differ  slightly  from 
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the  actual  highlight  color.  The  scalar  histogram  measurements  are  then  defined  simply  as 


/  =  MAX{ms)  over  all  p  (5.5) 

b  -  MAX(mh)  over  all  p  15.6) 

(  =  mj,  b  for  that  p  with  maximum  (5.7) 

M  =  [MAXinii,)  -  M/N(mi,)]  b  over  all p  for  which  >T  (5.8) 

where  the  threshold  is  given  by  T. 


The  lookup-table  approach  assumes  that  there  is  a  fundamental  relationship  between  histogram  dimen¬ 
sions  of  length,  width  and  intersection,  and  the  scene  parameters  of  roughness,  phase  angle  and  illumination 
mtensity.  This  relationship,  described  m  chapter  3.  relies  upon  physical  models  of  body  reflectance  and 
surface  reflectance,  and  how  they  interact  to  form  color  pixels.  The  model  does  not  make  assumptions  about 
specific  object  shape,  although  it  does  assume  a  broad,  continuous  distnbution  of  surface  normals.  At  a 
minimum,  the  method  requires  that  surface  normals  correspondmg  to  key  pomts  m  the  histogram — a.  b.  cl. 
c2.  and  d  from  figure  3.1 — are  visible  to  the  camera,  since  they  are  necessary  for  the  calculations  of  length, 
width  and  intersection.  The  lookup  table  approach  also  expects  that  the  relationship  between  histogram 
measurements  and  scene  parameters  is  one-to-one.  assuming  that  very  different  scene  parameters  cannot 
give  rise  to  similar  histogram  shapes.  Figure  5.1 1  demonstrates  that  the  relationship  is  mdeed  unique. 


5.7  Calculating  Roughness,  Phase  Angle,  and  Illumination  Intensity 


The  surface  in  figure  5,11  shows  lines  of  equal  roughness  within  the  L-W-I  space.  Once  length,  w'idth  and 
intersection  rreasurements  have  been  obtained  from  a  histogram,  the  problem  is  to  determme  which  line 
they  fall  on.  The  lines  shown  in  the  figure  come  from  the  lookup  table  values  and  are  fairly  close  together. 
However  it  is  unlikely  that  the  histogram  from  any  given  image  wfll  have  measurements  that  fall  exactly  on 
these  lines.  Therefore  some  sort  of  interpolation  is  needed. 

This  thesis  uses  polynomial  approximations  to  the  surface  in  L-W-I  space.  It  is  assumed  that  the 
roughness  c  can  be  approximated  as  an  «th  degree  polynomial  function  of  the  length,  width,  and  intersection 
measurements  of  the  histogram. 

a  s:  (5.9) 


2;  A  +  BI  +  Cw  +  Di  +  El~  +  Fw“  +  Gi*  +  H  Iw  +  Hi  +  Jwi  +  ... 
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Function 

R  squared 

cr  =/„(/.  H’,/) 

0.877 

=  g>,iLv.\i) 

0.939 

Bs  =  h„{L  H-.  i) 

0.712 

Figure  5.14;  Goodness-of-fit  for  polynomial  functions 

The  lookup  table  provides  the  means  for  calculating  the  coefficients  of  the  polynomial.  It  provides  almost 
500  sets  of  histogram  measurements  (see  hgtire  5.12)  and  the  associated  roughness  values.  Least  squares 
estimation  is  used  to  calculate  the  best  fit  nth  degree  polynomial  to  the  data.  A  fourth  degree  polynomial 
was  used  in  the  experiments  descnbed  here. 

Similarly,  the  phase  angle  and  illumination  intensity  are  also  approximated  as  polynomial  functions  of 
the  histogram  length,  width,  and  intersection. 

&p  -  gnU.W.l)  (5.10) 

Bs  ^  h„{Lw.i)  (5.11) 

Least  squares  is  used  to  calculate  the  coefficients  for  these  polynomials  also.  Since  the  length  measurement 
is  a  distance  in  RGB  space,  rather  than  a  percentage  of  the  length  of  the  body  reflection  cluster,  the  length 
measurement  was  first  divided  by  255  so  that  it  would  fall  in  approximately  the  same  range  as  the  width  and 
intersection  values  (roughly  between  0  and  1).  The  fit  of  the  least  squares  calculation  to  the  data  is  shown 
for  each  function  in  the  table  in  figure  5.14. 

Generating  the  lookup  table  is  obviously  very  time  consuming  (about  8  hours  on  a  SPARC  H)  since  it 
involves  calculating  almost  500  graphics  simulations.  However  this  and  the  coefficient  calculation  only  need 
to  be  done  once.  Given  the  histogram  of  an  object  with  unknown  parameters,  the  system  can  automatically 
separate  it  into  the  two  clusters  and  measure  their  dimensions  as  described  in  section  5.6.  The  polynomial 
calculation  can  then  be  applied  to  quickly  estimate  the  roughness,  phase  angle,  and  illumination  intensity'. 
This  algorithm  is  diagrammed  in  figure  5.15.  The  run-time  ponion  is  very  quick,  taking  less  than  3  seconds 
on  a  histogram  containing  about  3600  pixels.  If  the  histogram  has  already  been  split  into  clusters  in  the 
process  of  segmenting  the  image  [Klinker,  1988],  the  time  to  calculate  the  scene  parameters  is  less  than  1 
second. 

To  test  the  polynomial  approximations,  one  hundred  test  images  were  simulated  and  then  analyzed  by 
this  method.  These  test  images  were  noise-fi?e,  generated  with  the  same  idealized  camera  model  used  to 
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Figure  5.15:  Algorithm  for  calculating  roughness,  phase  angle,  and  illumination  intensity 

generate  the  lookup  table.  The  surface  roughness,  phase  angle,  and  illumination  intensity-  values  used  m  the 
lest  images  were  chosen  by  a  pseudo-random  number  generator.  The  values  were  constrained  to  lie  within 
the  ranges  used  in  the  lookup  table  (see  figure  5.12). 

The  calculated  values  of  cr.  Op.  andB^  were  compared  with  the  original  values  used  to  generate  the  image. 
In  almost  all  cases  the  calculated  values  were  very  close  to  the  original  ones.  However,  for  2  of  the  100  test 
cases,  the  values  were  very  obviously  wrong.  For  example,  a  negative  value  of  roughness  or  illumination 
intensity  is  clearly  unreasonable.  Fortunately  bad  values  are  easily  detected  by  checking  to  see  if  recovered 
values  are  within  the  allowable  range.  Recovered  values  outside  that  range  indicate  that  a  different  method 
should  be  used  to  recover  the  scene  parameters. 

For  example,  a  lower  degree  polynomial  approximation  could  be  used  for  these  cases.  Although  the 
fourth  degree  polynomial  is  better  at  approximating  the  function  over  its  whole  range,  it  may  deviate 
somewhat  at  the  extremes  of  the  function.  In  fact  the  2  problem  cases  occin  when  the  roughness  is  very 
low  (between  1  and  2  degrees)  and  the  phase  angle  is  very  large  (greater  than  75  degrees).  The  problem 
disappears  if  a  third  degree  polynomial  is  used  instead,  although  the  overtdl  error  on  all  cases  is  slightly 
higher. 

The  table  in  figure  5.16  shows  the  results  for  the  remaining  98  cases  where  the  fourth  degree  polynomial 
produced  believable  results.  It  shows  the  average  error  in  recovering  the  parameters,  and  also  reiterates  the 
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Parameter 

Average  Error 

Resolution  of  Table 

Roughness 

1.20= 

2= 

Phase  Angle 

4.40= 

10= 

Intensity' 

8.18  % 

10% 

Cases  Considered 

98/100 

Figure  5.16;  Results  on  noise-free  data 

step  sizes  used  in  the  table.  The  errors  are  lower  than  the  table  resolution,  showing  the  interpolation  method 
is  fairly  effective. 

The  results  for  calculating  roughness  and  phase  angle  are  veiy'  good.  They  show  that  these  non-color 
parameters  may  be  calculated  with  reasonably  high  accuracy  just  by  considering  the  shape  of  the  color 
histogram.  The  error  in  calculating  illumination  intensity’  result  is  a  bit  higher,  although  it  still  provides 
a  useful  estimate.  This  error  is  not  too  surprising,  since  the  R-squared  fit  of  the  function  to  calculate 
illumination  intensity'  is  somewhat  worse  than  for  the  other  two  functions  (see  figure  5.14). 


5.8  Calculating  Illumination  Chromaticity 

As  has  been  discussed  in  section  3.2.1  the  highlight  cluster  may  be  somewhat  skewed  from  the  direction 
of  the  highlight  color.  The  skew  is  particularly  pronounced  for  large  phase  angles  and  when  the  surface  is 
rough.  These  two  factors  determine  how  much  the  body  reflection  changes  over  the  area  of  the  highlight. 
Figure  5.17  shows  how  the  skew  varies  as  a  function  of  roughness  and  phase  angle.  The  height  of  the 
graph  is  the  angle  (in  degrees)  between  a  vector  fitted  to  the  highlight  cluster  and  the  vector  defined  by  the 
illumination  color. 

Therefore,  if  the  analyzing  program  knows — or  can  calculate — the  surface  roughness  and  the  imaging 
geometry,  it  can  in  turn  calculate  the  amount  of  highlight  skewing.  Once  the  skew  is  known,  its  effect  can 
be  subtracted  from  the  direction  of  the  highlight  cluster  to  give  the  true  color  of  the  illumination. 

The  previous  section  showed  how  to  estimate  the  roughness  and  phase  angle  from  the  color  histogram. 
These  estimates  will  now  be  used  to  estimate  the  skewing.  A  similar  lookup  table  approach  is  used.  When 
the  simulations  are  performed  to  calculate  length,  wadth,  and  intersection  for  the  range  of  scene  parameters 
given  in  figure  5.12,  the  skewing  of  the  highlight  is  also  calculated.  In  the  graphics  simulation  the  correct 
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Figure  5.17;  Variation  in  cluster  skew  with  changes  in  roughness  and  phase  angle 


illumination  color  is  obviously  known  in  advance,  so  the  angle  between  this  color  and  the  vector  fitted  to 
the  highlight  cluster  is  calculated  and  stored  in  the  lookup  table  along  with  the  values  for  length,  width,  and 
intersection.  Then  a  polynomial  function  is  used  to  calculate  the  skew  angle  as  a  function  of  roughness, 
phase  angle,  and  intensity: 

Skew  ^  A„{cr,6p,Bs)  (5.12) 

The  coefficients  of  the  «th  degree  polynomial  function  are  calculated  using  a  least  squares  fit  to  the  data  in 
the  lookup  table.  A  third  degree  polynomial  was  tested  and  gave  an  R-sqtiaied  fit  of  0.992. 


Once  the  skew  has  been  calculated,  the  normalized  highlight  color  is  calculated  from  the  measured 
direction  ds  by  the  constraints 


cl  'ds  =  cosiSkew) 

(5.13) 

cl  -db  =  cosiSkew  +  arccosCdfc  •  ds)) 

(5.14) 

idsXdb)  =  0 

(5.15) 

Equation  5.13  simply  says  that  the  angle  between  the  highlight  color  cl  and  the  measured  cluster  direction 
ds  is  Skew.  Equation  5.14  indicates  that  the  angle  between  the  highlight  color  cl  and  the  body  reflection 
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Average  Error 

1.73° 

Average  Skew 

8.63° 

Minimum  Value 

0.01° 

Maximum  Value 

27.5° 

Number  of  Skews  >  1  ° 

69 

Cases  Considered 

98/100 

Figtire  5.18;  Resixlts  for  calculating  skew 

direction  dt  is  Skew  plus  the  angle  between  the  cluster  direction  and  the  body  reflection  direction  (the 
skewed  highlight  cluster  direction  must  fall  between  the  true  highlight  color  and  the  body  reflection  color). 
Equation  5.15  states  that  all  three  vectors  must  fall  in  a  plane. 

Obviously,  if  the  polynomial  functions  described  in  section  5.7  produce  unreasonable  estimates  of  the 
roughness,  phase  angle,  or  intensity,  there  is  little  point  in  plugging  them  into  the  equation  for  calculating 
Skew.  Therefore,  for  the  2  test  cases  that  were  rejected,  the  program  did  not  attempt  to  calculate  the 
illumination  color.  For  the  remaining  98  test  cases,  the  skew  angle  was  used  to  calculate  the  illumination 
color.  The  results  are  shown  in  the  table  in  figure  5.18.  The  error  in  estimating  skew  is  the  difference — in 
degrees — ^between  the  correct  skew  angle  and  the  calculated  skew  angle.  The  correct  skew  angle  is  easily 
calculated  from  the  illumination  color  that  was  used  to  generate  the  simulation.  The  table  shows  the  average 
error  over  the  98  cases  considered.  It  also  shows  the  minimum,  maximum  and  average  of  the  actual  skew 
values.  For  69  of  the  100  test  images,  the  scene  parameters  were  such  that  the  skew  exceeded  one  degree. 

As  an  example,  one  of  the  test  simulations  is  shown  in  color  figure  A.4.  This  is  a  red  cylinder  under 
white  light,  so  ci  =  [0.58,0.58,0.58].  The  surface  roughness  has  a  12.06°  standard  deviation  of  the  facet 
angles;  the  phase  angle  separating  the  camera  and  light  source  with  respect  to  the  object  is  63.30°;  and  the 
illumination  intensity  is  90%  of  maximum. 

The  histogram  associated  with  this  image  is  shown  in  figure  5.19.  The  graph  shows  the  projection  of 
the  color  histogram  into  the  red-green  plane.  The  program  automatically  divided  the  histogram  into  body 
reflection  and  surface  reflection  clusters.  The  vector  fitted  to  the  cluster  is  dj  =  [0.81,0.45,0.37].  The  length 
of  the  highlight  cluster  was  measmed  as  74.1;  the  width  of  the  highlight  cluster  was  measured  as  0.47  (the 
highlight  cluster  extends  over  slightly  less  than  half  the  body  cluster);  the  intersection  of  the  two  clusters 
was  measured  as  0.51  (the  brightest  point  in  the  highlight  cluster  was  projected  onto  the  body  cluster  just 
above  the  halfway  point).  These  measurements  was  all  obtained  using  the  vector  ds  to  calculate  the  amount 
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Figure  5.19;  Histogram  of  simulated  image 
of  body  reflection  and  surface  reflection. 

The  direction  fitted  to  the  highlight  cluster  is  significantly  skewed  away  from  the  direction  of  the  actual 
illumination  color.  It  is  significantly  reddened  compared  to  the  achial  illumination  color,  and  so  would  be  a 
poor  estimate  of  the  illumination  color.  It  would  also  yield  an  inaccurate  estimate  of  the  object  color  when 
the  influence  of  the  illumination  color  is  divided  out  of  the  body  reflection  color. 

Applying  equations  5.9, 5.10  and  5.1 1  to  the  length,  width  and  intersection  measurements,  the  program 
estimated  the  roughness  value  as  1 1 .99°,  the  phase  angle  as  67.05°,  and  the  illumination  intensitj'  as  89%  of 
maximum.  Applying  equation  5.12  to  these  values  for  roughness,  phase  angle,  and  illumination  intensity, 
the  skew  between  the  highlight  cluster  direction  and  the  actual  illumination  color  was  estimated  to  be  18.75°. 
Solving  the  constraint  equations  5.13  through  5.15  produced  an  estimate  of  the  light  source  chromaticity  as 
Cs  =  [0.59,0.57,0.57].  This  is  very  close  to  the  original  white  color. 

The  results  for  this  example  are  summarized  in  the  table  in  figure  5.20.  The  recovered  parameters  were 
also  used  to  generate  a  a  new  simulation  picture,  shown  in  color  figure  A.5.  The  difference  between  this 
image  and  the  original  image  in  figure  A.4  is  very  subtle.  The  full  algorithm,  including  the  technique  for 
estimating  the  illumination  color,  is  diagrammed  in  figure  5.21. 

The  object  reflectance  used  to  simulate  the  cylinder  cl  is  [0.94,0. 29, 0.16],  If  the  highlight  direction 
vector  ds  is  assumed  to  be  the  illumination  color,  then  removing  its  influence  from  the  body  color  yields  an 
estimate  of  [0.83,0.47,0.31]  for  the  object  reflectance.  However,  the  corrected  estimate  of  the  illumination 
color  obtained  by  calculating  the  highlight  skew  yields  an  estimate  of  [0.94,0.30,0.17]  for  the  object’s 
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reflectance,  much  closer  to  the  actual  reflectance  used  to  generate  the  image.  Thus,  accounting  for  the  skew 
of  the  highlight  leads  to  improved  estimates  of  object  reflectance  for  color  constancy. 

Calculating  the  illumination  color  also  allows  the  system  to  generate  intrinsic  images  of  body  reflection 
and  surface  reflection,  as  shown  in  fKlinker,  1988].  The  intrinsic  body  reflection  image  is  calculated  by 
projecting  all  pixels  along  the  direction  of  the  highlight  onto  the  body  reflection  vector.  Sunilarh,  the 
intrinsic  surface  reflection  comes  from  projecting  all  pixels  along  the  direction  of  the  body  color  onto  the 
highlight  direction  vector.  Given  that  each  pixel  in  the  original  image  has  been  separated 

p  =  rrisJs  +  mt,^b 

Then  each  pixel  in  the  resulting  intrinsic  body  reflection  image  is  just 

Pb  =  ntblb 

while  the  intrinsic  surface  reflection  pixels  are  calculated 

Ps^mJs 


However  if  the  highlight  color  has  been  calculated  incorrectly  by  failing  to  account  for  the  skew  of  the 
highlight  cluster  towards  the  body  reflection  color,  those  pixels  in  the  highlight  will  be  projected  too  far 
down  the  body  reflection  vector,  making  them  appear  too  dark.  This  situation  is  shown  in  color  figure  A.6. 
The  intrinsic  images  were  calculated  using  the  highlight  direction  [[0-81,0.45,0.37]],  as  measured  in  the 
histogram  before  correcting  for  the  skew.  The  cylinder  on  the  left  is  the  intrinsic  body  reflection  while  the 
cylinder  on  the  right  is  the  intrinsic  surface  reflection.  Because  this  highlight  cluster  is  skewed  away  from 
the  white  color  of  the  original  illumination,  the  intrinsic  body  reflection  image  has  a  dark  area  where  the 
highlight  was.  The  intrinsic  surface  reflection  image  is  much  redder  than  the  actual  illumination  color. 

However,  when  the  skew  of  the  highlight  is  accounted  for,  the  corrected  illumination  estimate  of 
[0.59,0.57,0.57]  can  be  used  to  calculate  the  intrinsic  images.  Now  the  pixels  in  the  highlight  will  be 
projected  correctly  onto  the  body  reflection  vector.  The  result  is  shown  in  color  figure  A. 7.  The  intrinsic 
body  reflection  image  on  the  left  appear  correctly  shaded,  and  the  intrinsic  surface  reflection  image  on  the 
right  correctly  shows  that  the  highlight  color  is  white. 

This  chapter  has  shown  that  the  method  developed  works  very  well  on  perfect  images  generated  without 
camera  limitations.  Any  practical  method  will  have  to  work  on  real  images  and  must  deal  with  noise  and 
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Simulated  Image  cl  =  [0.58,0.58,0.58]  <7=12.06'’  0p  =  63.3(r  B,  =  909c 

I  Histogram  Measurements  £^  =  [0.81,0.45,0.37]  1  =  74.1  h’ =  0.47  /  =  0.5I 

'  Recovered  Parameters  cl  =  [0.59,0.57.0.57]  £7=11.99'’  gp  =  67.05'’  Bj  =  89% 


Figure  5.20:  Results  for  test  image 


Off-line: 


Run-time: 


Figure  5.21:  Algorithm  for  recovering  scene  parameters 
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other  camera  problems.  The  next  chapter  will  show  how  the  method  developed  here  may  be  extended  for  a 
camera  model  that  incorporates  some  limitations  found  in  real  cameras. 
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Chapter  6 


Extending  the  Analysis  to  a  Realistic  Sensor 
Model 


Chapter  5  described  how  to  estimate  scene  parameters  from  a  color  histogram  if  an  ideal  sensor  is  used. 
Specifically,  it  assumed  that  the  camera  used  to  view  objects  is  noise-free  and  linear  over  the  range  of  all 
possible  intensity  values.  The  possibility  of  clipping  at  some  maximum  camera  value  was  not  considered  in 
chapter  5. 

However  these  are  not  very  realistic  assumptions.  Some  forms  of  noise  exist  in  all  real  camera  systems. 
And,  while  linearity  of  response  over  a  limited  range  is  common  in  many  cameras,  all  real  cameras  and 
digitizers  have  some  maximum  response.  Clipping  at  the  maximum  value  is  particularly  common  in  pictures 
with  highlights  since  they  are  so  much  brighter  than  other  parts  of  the  scene. 

Since  this  thesis  aims  to  analyze  pictures  with  highlights,  and  since  all  real  imaging  systems  have  noise, 
it  will  be  necessary  to  deal  with  these  issues  in  order  to  make  the  algorithm  work  on  real  images.  Section  6. 1 
will  describe  how  the  aJgorithmhas  been  adapted  to  deal  with  noise,  whil^  section  6.2  win  describe  a  method 
for  dealing  with  the  problem  of  clipping. 
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6.1  Modeling  Camera  Noise 


Rea!  cameras  present  many  limitations  for  taking  accurate  images.  CCD  cameras,  which  are  used  by  most 
vision  researchers,  present  many  problems  such  as  dark  current,  shot  noise,  amplifier  noise,  line  jitter,  etc. 
[Healey  and  Kondepudy,  1991],  Some  of  these  problems,  such  as  shot  noise,  are  an  inherent  property  of 
CCD  cameras  and  cannot  be  removed  even  with  high-quality  manufacturing. 

This  thesis  models  camera  noise  as  having  a  Gaussian  distribution  with  a  mean  of  zero.  The  standard 
deviation  of  camera  noise  Cc  was  modeled  as  1.275.  The  camera  is  assumed  to  measure  eight  bits  per  pixel, 
giving  a  range  from  0  to  255.  Thus  the  ratio  of  maximum  pixel  value  to  standard  deviation  of  noise  is  then 
200. 

Since  the  method  described  in  this  thesis  estimates  scene  parameters  by  analyzing  color  histograms,  it 
must  consider  the  effect  of  noise  on  the  histogram's  appearance.  Under  the  Gaussian  noise  model,  each 
pixel  may  be  viewed  as  occupying  a  probability  cloud  in  color  space.  This  cloud  represents  the  pixel’s  idea! 
color  value  plus  some  uncertainty  due  to  noise.  The  size  of  the  cloud  depends  upon  the  standard  deviation 
of  the  noise.  If  the  population  of  pixels  is  large,  the  cloud  will  be  filled  according  to  the  noise  distribution. 
For  a  Gaussian  distribution,  the  modal  pixel  will  coincide  with  the  ideal  pixel  value,  with  two-thirds  of  the 
pixels  falling  within  one  standard  deviation  of  the  ideal  value. 

One  of  the  effects  of  noise  on  the  histogram  is  that  it  will  make  the  body  reflection  cluster  thicker.  A 
uniformly  colored  object  imaged  with  an  ideal  camera  would  have  a  body  reflection  cluster  that  is  a  line  only 
one  pixel  thick.  Noise  added  to  ideal  pixels  will  correspond  to  some  pixels  that  are  displaced  away  from  the 
ideal  line.  This  will  result  in  a  thicker  body  reflection  cluster.  However,  even  in  the  ideal  case  the  algorithm 
needed  a  threshold  for  distinguishing  highlight  pixels  from  pixels  on  the  body  reflection  cluster,  although 
this  threshold  could  be  arbitrary  (see  section  5.6).  With  an  explicit  noise  model,  the  threshold  should  be 
based  on  the  standard  deviation  of  the  camera  noise.  The  threshold  T  was  chosen  to  be  4crc,  so  over  99%  of 
the  noisy  body  reflection  pixels  will  fall  within  the  body  reflection  cluster. 

Noise  added  to  body  reflection  pixels  will  also  push  some  pixels  up  or  down,  along  the  direction  of 
increasing  amounts  of  body  reflection.  Noise  added  to  those  pixels  which  are  at  the  tip  of  the  body  reflection 
cluster  (point  b  in  figure  3.1)  will  result  in  a  cluster  that  is  slightly  longer.  This  will  affect  the  measurement 
of  the  length  of  the  body  reflection  cluster  b  and  the  subsequent  calculation  of  the  object’s  albedo  B(,  (see 
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section  5.2).  Fortunately  the  effect  is  very  small  compared  to  the  total  length  of  the  body  refieciion  cluster, 
so  it  will  be  ignored. 

Noise  added  to  pixels  in  the  highlight  cluster  will  also  displace  those  pixels,  causing  the  highlight  cluster 
to  be  slightly  wider  and  longer  than  for  the  noise-free  case.  Unlike  noise  added  to  the  body  reflection  cluster, 
this  effect  must  be  considered  carefully,  since  the  estimation  method  uses  small  differences  in  the  width  of 
the  cluster  to  predict  differing  values  of  roughness.  Therefore  in  dealing  with  noise  in  the  highlight  pixels  a 
two  pronged  attack  is  used:  the  noise  is  explicitly  accounted  for  when  generating  the  lookup  tables;  and  an 
attempt  is  made  to  remove  some  noise  when  analyzing  each  histogram. 

When  the  lookup  tables  were  generated  for  chapter  5.  a  computer  program  simulated  noise-free  images 
and  measured  the  resulting  histograms.  Now,  for  the  noisy  camera  model,  images  are  simulated  again,  this 
time  with  pseudo-random  Gaussian  noise  added.  The  resulting  histograms  are  analyzed,  and  new  values  of 
length,  width  and  intersection  are  recorded  in  the  lookup  table.  These  measurements  will  reflect  the  slight 
increases  in  the  length  and  width  of  the  two  histogram  clusters. 

Furthermore,  in  measuring  the  histograms  local  smoothing  is  used.  Each  pixel  from  the  simulated  image 
is  averaged  with  its  four  nearest  neighbors  before  being  plotted  in  the  histogram.  For  pixels  falling  in  the 
highlight  or  at  its  edge,  this  will  have  the  effect  of  smoothing  out  the  highlight  somewhat,  but  will  also  tend  to 
average  out  the  noise.  The  effect  of  highlight  smoothing  on  the  length,  width  and  intersection  measiuements 
will  also  be  recorded  in  the  lookup  table. 

The  same  techniques  as  described  in  section  5.7  are  used  to  fit  polynomials  to  the  measurements  in  the 
new  table.  In  analyzing  noisy  histograms,  any  pixels  that  fall  outside  the  plane  defined  by  the  vectors  fitted 
to  the  clusters,  and  d*,  are  projected  into  that  plane.  Furthermore  the  body  reflection  vector  is  fitted 
slightly  differently  since  it  will  now  contain  noise.  The  pixel  furthest  from  the  line  drawn  to  the  brightest 
pixel  is  still  used,  but  it  is  an  initial  estimate.  All  pixels  that  are  no  further  than  the  threshold  T  from  that 
line  are  clumped  together  and  the  best  fit  line  to  that  cluster  becomes  the  new  body  reflection  estimate.  The 
process  is  repeated  tmtil  the  new  vector  estimate  is  within  one  tenth  of  a  degree  of  the  previous  vector,  or 
ten  iterations  have  taken  place. 

The  table  in  figure  6.1  presents  the  results  when  the  method  is  tested  on  100  simulated  images  with 
pseudo-random  Gaussian  noise  added.  The  algorithm  is  the  same  as  the  one  presented  in  figure  5.21 .  Only 
the  lookup  tables  have  been  changed,  to  reflect  the  noise  model  of  the  camera.  Comparing  the  noise-free 
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Parameter 

Average  Error 

Resolution  of  Table 

Roughness 

1.08“ 

2“ 

:  Phase  Angle 

4.56“ 

10“ 

Intensity 

6.85  % 

10% 

Skew 

1.39“ 

I 

Cases  Considered 

99/100 

Figure  6.1;  Results  on  noisy  data  (cTc  =  1 .275) 


Parameter 

Average  Error 

Resolution  of  Table 

Roughness 

0.97“ 

2- 

Phase  Angle 

7.07“ 

10“  1 

Intensity 

7.49  % 

10%  i 

!  Skew 

1.29“ 

- 

1  Cases  Considered 

98/100 

Figure  6.2:  Results  on  noisier  data  (cr^  =  2.55) 


results  in  figures  5.16  and  5.18  with  the  results  here,  we  see  that  the  algorithm  acttially  did  slightly  better 
with  noisy  data,  except  when  calculating  phase  angle.  This  is  a  surprising  result,  but  it  is  caused  by  the 
averaging  that  is  done  in  on  the  noisy  data,  which  has  the  effect  of  smoothing  the  histogram  slightly. 

A  second  experiment  was  performed  to  see  how  the  system  handles  a  larger  amount  of  noise.  Accordingly, 
the  tables  of  length,  width,  and  intersection  were  recomputed  for  a  camera  with  twice  the  noise  level 
{(Tc  =  2.55).  The  results  are  shown  in  figure  6.2.  They  show  that  the  estimates  of  phase  angle  are 
significantly  worse  than  for  the  low  noise  case,  although  the  error  is  still  less  than  the  resolution  of  the  table. 
The  errors  in  estimating  roughness,  illumination  intensity,  and  skew  angle  are  affected  only  slightly.  This 
shows  that  the  method  does  not  suffer  significantly  if  the  noise  of  the  camera  is  larger,  provided  that  noise 
has  been  modeled  when  the  lookup  tables  are  calculated. 

If  the  noise  if  not  properly  modeled  when  calculating  the  lookup  tables,  the  method  does  not  work  well 
at  all.  For  example,  if  the  larger  noise  value  (<7^  =  2.55)  is  used  to  simulate  the  test  images,  but  the  smaller 
noise  value  (cr<.  =  1.275)  is  used  to  generate  the  lookup  tables  and  compute  the  polynomial  functions,  the 
method  produces  implausible  answers  26%  of  the  time,  and  the  errors  in  computing  the  scene  parameters 
are  much  higher. 
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6.2  Modeling  Camera  Clipping 


Like  noise,  clipping  is  an  inherent  property  of  real  cameras  since  all  real  cameras  have  some  maximum  value 
that  they  can  measure.  Many  real  images  with  highlights  suffer  from  clipping  since  highlights  are  often  verv 
bright.  When  taking  real  pictures,  it  is  usually  possible  to  adjust  the  aperture,  light  level,  or  exposure  time  to 
a  certain  extent.  However,  highlights  on  shiny  objects  are  so  much  brighter  than  sunounding  non-highlight 
areas,  that  if  the  camera  is  adjusted  to  measure  highlights  properly,  the  non-highlight  areas  will  be  too  dark 
to  rely  upon  theii  values. 

Therefore  any  algorithm  that  hopes  to  work  on  real  pictures  with  highlights  should  take  clipping  into 
account.  The  effect  of  clipping  on  the  color  histogram  is  easy  to  model.  Any  pixels  that  would  have  exceeded 
the  maximum  value  (255  for  an  8  bit  camera)  will  be  held  to  that  value.  As  a  result,  the  highlight  cluster  will 
not  be  as  long  as  it  otherwise  would  have  been.  Clipping  usually  occurs  when  the  surface  roughness  has  a 
standard  deviation  of  facet  angles  of  3°  or  less. 

In  analyzing  color  histograms,  clipping  is  easy  to  detect.  If  a  pixel  has  the  maximum  value  in  any  band,  it 
was  almost  certainly  clipped.  However,  dealing  with  the  clipping  presents  a  greater  problem  since  the  effect 
upon  the  length  may  be  dramatic.  Fortunately  the  width  measurement  is  not  affected  at  all,  as  long  as  there 
are  some  pixels  at  the  edge  of  the  highlight  that  do  not  saturate  the  camera.  The  intersection  measurement 
can  be  affected,  since  it  relies  upon  the  projection  of  the  brightest  highlight  pixel.  In  the  case  of  clipping,  the 
brightest  non-saturated  pixel  is  used  to  calculate  the  intersection  point.  Fortunately,  the  difference  is  usually 
very  small.  This  is  because  the  highlight  cluster  is  usually  very  narrow  at  the  point  where  it  is  clipped. 

The  length  measurement  is  an  important  parameter  in  calculating  the  scene  parameters,  along  with  the 
width  and  intersection  measurements.  In  order  to  calculate  the  three  unknown  values  of  surface  roughness, 
phase  angle  and  illumination  intensity,  three  known  histogram  measurements  are  needed.  If  the  length 
measurement  cannot  be  used,  it  will  have  to  be  replaced  with  some  other  measurement. 

Therefore  the  length  measurement  is  replaced  with  another  measurement  of  the  cluster  shape.  The  width 
measurement  and  the  length  measurement  are  a  way  of  characterizing  the  shape  of  the  highlight  cluster. 
The  algorithm  needs  to  capture  this  shape  since  it  is  related  to  the  relative  distributions  of  body  and  surface 
reflection,  which  in  turn  are  determined  by  the  scene  parameters  we  want  to  estimate. 


When  the  length  measurement  is  untrustworthy  due  to  clipping,  the  method  will  then  use  a  second  width 
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Figure  6.3:  Purpose  of  second  width  measurement 


measurement.  The  first  width  measurement  tells  how  wide  the  highlight  cluster  is  at  the  threshold  distance  T 
from  the  body  reflection  cluster.  The  second  width  measurement  was  chosen  to  show  how  wide  the  highlight 
cluster  is  at  a  distance  of  ST  from  the  body  reflection  cluster.  This  measurement  will  allow  the  system  to 
distinguish  between  cases  where  the  highlight  cluster  would  have  extended  only  slightly  beyond  the  color 
cube  boundary,  and  cases  where  the  highlight  cluster  would  have  been  much  longer.  These  two  cases  are 
sketched  in  figure  6.3. 

When  the  lookup  tables  are  calculated,  the  length  measurement  is  recorded,  and  also  an  indication  as  to 
whether  or  not  it  is  clipped.  At  the  same  time,  the  second  width  measurement  is  recorded,  as  well  as  the  first 
width  and  intersection  measurements  previously  described.  After  the  lookup  table  is  generated  two  sets  of 
polynomial  functions  are  fitted:  one  for  analyzing  histograms  that  are  not  clipped,  and  one  set  for  histograms 
that  are.  If  the  second  width  measurement  is  referred  to  as  W2  then  the  scene  parameters  may  be  calculated 
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Parameter 

Average  Error 

Resolution  of  Table 

i  Roughness 

1.00“ 

2“ 

Phase  Angle 

6.07“ 

10“ 

Intensity 

6.76  9c 

10  9c 

i  Skew 

1.69“ 

- 

1  Cases  Considered 

96/100 

Figxire  6.4:  Results  on  noisy  data  when  clipping  is  present 


Parameter 

Noise-free 

Noisy 

Noisy  and  Clipped 

Roughness 

0 

0 

1.08“ 

1.00“ 

i  Phase  Angle 

4.40“ 

4.56“ 

6.07“ 

!  Intensity 

8.18% 

6.85  % 

6.76  % 

Skew 

1.73“ 

1.39“ 

1.69“ 

1  Cases  Considered 

98 

99 

96 

Figure  6.5:  Errors  for  three  camera  models 
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if  the  histogram  is  not  clipped 


if  the  histogram  is  clipped 


(6.1) 


(6.2) 


Since  there  are  fewer  data  points  in  the  lookup  table  for  the  clipped  case,  a  third  degree  pol>T3omiaI  is  fitted 
for  functions/2,  g2,  and  /12. 


The  results  obtained  by  applying  this  technique  to  100  simulated  test  images  with  Gaussian  noise  added 
((7c  =  1.275)  are  shown  in  the  table  in  figure  6.4.  The  results  here  are  similar  to  those  obtained  without 
clipping,  although  there  are  more  problem  cases  that  had  to  be  eliminated  from  consideration.  Results  from 
the  noise-free,  noisy,  and  noisy  with  clipping  algorithms  are  all  shown  in  figure  6.5. 
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Chapter  7 


Application  of  the  Algorithm  to  Real  Images 

Chapter  5  presented  a  method  for  analyzing  color  histograms  assuming  an  ideal  imaging  system.  Chapter  6 
showed  how  that  method  may  be  modified  in  a  systematic  way  for  a  more  realistic  camera  system.  This 
chapter  will  present  some  results  from  applying  the  algorithm  developed  in  chapter  6  to  real  images. 

The  real  images  in  this  thesis  were  taken  in  the  Calibrated  hnaging  Laboratory  (CIL)  at  Carnegie  Mellon 
University.  The  CEL  provides  a  controlled  setting  for  taking  high-quality  images.  It  also  allows  for  a  careful 
analysis  of  camera  issues  that  may  cause  problems  lor  vision  algorithms.  The  equipment  and  capabilities 
are  described  in  [Willson  and  Shafer,  1992]. 


7.1  Additional  Problems  with  Real  Cameras 


Although  chapter  6  modeled  some  fundamental  limitations  of  real  cameras,  including  noise  and  clipping, 
other  problems  that  are  commonly  encountered  in  imaging  were  not  modeled.  These  problems  include 
nonlinearity,  color  imbalance,  chromatic  aberration,  and  blooming.  Some  problems  such  as  linearity  and 
color  balance  are  easily  fixed  in  software  or  with  simple  hardware.  Others  are  more  difficult  to  handle. 

The  degree  of  each  of  these  problems  is  very  specific  to  the  particular  camera  system  used.  This  makes 
them  very  difficult  to  model  in  a  general  way.  Some  systems  may  have  been  designed  to  reduce  or  even 
eliminate  a  specific  problem,  such  as  nonlinearity. 


Ill 
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7.1.1  Nonlinearity  and  Color  Imbalance 

The  methods  for  analyzing  histograms  developed  in  this  thesis  require  that  the  camera  have  a  linear  response. 
This  is  because  they  measure  specific  distances  in  the  color  space  and  assume  that  these  distances  are 
proportional  to  the  amounts  of  body  reflection  and  surface  reflection  observed  on  the  object. 

For  example,  the  intersection  measurement  described  in  section  3.2.4  is  assumed  to  record  an  estunate 
of  the  amount  of  body  reflection  at  the  brightest  point  in  the  highlight.  The  ratio  between  the  body  reflection 
at  that  point  and  the  maximum  amount  seen  anywhere  on  the  object  is  proportional  to  the  cosine  of  the 
angle  between  the  light  source  and  the  perfect  specular  direction.  However  if  the  camera  is  not  linear  in  its 
response,  that  proportionality  w-ill  no  longer  hold. 

Fortunately,  linear  CCD  cameras  are  commonly  available.  In  fact  CCD  cameras  are  inherently  linear 
since  they  produce  a  voltage  that  is  proportional  to  the  number  of  photons;  the  problem  is  that  some 
models  are  designed  to  drive  displays  and  have  added  circuitry  that  makes  them  nonlinear.  In  this  case,  the 
nonlinearity  may  be  measured  using  a  standard  reference  chart  and  corrected  with  a  look’.p  table  [Klinker, 
1988],  [Novak  et  al.,  1990].  The  camera  used  to  take  the  real  images  in  this  chapter  already  has  a  linear 
response,  so  no  correction  is  necessary. 

Another  common  problem  with  CCD  cameras  is  color  imbalance.  CCDs  are  typically  much  less  sensitive 
to  blue  wavelengths  than  they  are  to  green  or  red  wavelengths.  In  addition,  incandescent  hghts  are  usually 
chosen  for  taking  pictures  in  the  CIL.  This  is  for  three  main  reasons;  incandescent  lighting  is  somewhat 
closer  to  the  point  source  model  than  most  fluorescent  bulbs;  the  intensity  of  an  incandescent  light  is  easily 
controlled  with  a  variable  voltage  supply;  and  fluorescent  lights  have  a  very  spiky  spectral  power  distribution 
that  can  cause  interference  patterns  in  the  lens. 

Incandescent  lights  are  also  strong  in  the  long  wavelengths  and  weak  in  the  short  wavelengths.  Therefore 
when  incandescent  lights  are  used  wiih  CCD  cameras,  the  blue  response  is  extremely  poor.  If  color  pictures 
are  taken  without  some  form  of  color  correction,  blue  objects  will  be  extremely  dark,  and  white  objects  will 
appear  yellowish. 

A  simple  solution  would  be  to  scale  the  blue  band  of  all  color  images  by  an  appropriate  constant.  The 
problem  with  this  approach  is  that  while  scaling  up  the  blue  signal,  it  would  also  scale  up  the  noise.  Therefore 
it  would  be  better  to  increase  the  signal  in  some  other  way,  such  that  the  values  in  the  blue  band  are  as 
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reliable  as  those  in  the  red  and  green  bands. 

In  the  CIL,  two  different  approaches  have  been  used  to  increase  the  blue  response.  In  both  cases,  color 
pictures  are  taken  with  a  gray-scale  camera  and  a  filter  wheel  to  produce  the  red,  green,  and  blue  bands.  This 
allows  adjustments  to  be  made  individually  for  each  band.  In  the  case  of  a  camera  with  a  fixed  exposure 
time,  the  aperture  is  changed  between  images  of  the  bands,  so  that  the  aperture  can  be  much  wider  to  admit 
more  light  for  the  blue  band. 

A  white  test  card  is  used  to  select  the  appropriate  aperture  settings  for  each  band.  The  aperture  on  the 
CDL  cameras  is  under  computer  control  with  a  motorized  lens  that  has  2700  steps  for  aperture  control.  This 
aperture  control  method  has  the  disadvantage  that  the  depth  of  field  changes  slightly  between  bands,  but  this 
effect  is  not  noticed  in  scenes  with  a  small  range  of  depths.  Figures  A.l  and  A.2  shown  in  chapter  3  were 
taken  with  this  technique. 

A  different  technique  is  used  for  cameras  that  have  a  variable  exposure  time.  Again,  a  white  test  card  is 
used,  although  in  this  case  the  test  images  are  used  to  select  the  appropriate  exposure  times.  The  exposure 
time  can  be  as  short  as  0.0  seconds  and  as  long  as  6,553.6  seconds,  with  a  resolution  of  0.1  second.  In  the 
CBL,  the  exposure  tune  for  the  blue  band  is  typically  ten  times  longer  than  the  red  or  green  exposures.  The 
images  shown  in  this  chapter  were  taken  with  this  latter  technique. 


7.1,2  Chromatic  Aberration 

Another  common  problem  in  real  images  is  chromatic  aberration.  This  is  an  inherent  property  of  lenses, 
because  the  index  of  refraction  of  optical  components  varies  as  a  function  of  wavelength.  Chromatic 
aberration  can  be  partially  compensated  for  during  manufacturing  by  using  pairs  of  lens  elements  with 
offsetting  dispersion  functions.  Unfortunately  this  t5rpe  of  compensation  is  usually  only  calculated  for  two 
wavelengths  (red  and  blue)  and  for  a  small  number  of  points  in  the  image  field. 

Chromatic  aberration  has  two  main  effects:  the  different  color  bands  will  have  slightly  different  points 
of  best  focus;  and  the  different  bands  will  have  slightly  different  magnifications.  A  third  effect,  which  is 
observed  in  real  lenses,  is  a  lateral  displacement  of  the  images  between  bands,  caused  by  slight  misalignment 
of  the  lens  components.  These  effects  will  cause  noticeable  shifts  in  the  color  of  affected  pixels.  Experience 
in  the  CIL  has  shown  that  even  high  quality  lenses  suffer  from  chromatic  aberration  that  is  readily  apparent 
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Without  Chromatic  Aberration 


With  Chromatic  Aberration 


Figure  7.1:  Sketch  of  the  effects  of  chromatic  aberration  on  a  simple  histogram,  as  described  by  Willson  and 
Shafer,  1991 


in  histogram  analysis.  For  example,  the  histogram  of  a  black  grid  on  a  white  background  would  be  expected 
to  show  only  white  pixels,  black  pixels,  and  gray  pixels  where  an  edge  in  the  image  fell  somewhere  in  the 
middle  of  a  pixel.  When  chromatic  aberration  is  present,  some  of  the  pixels  at  the  edges  will  be  shifted 
towards  red  whereas  others  will  be  shifted  towards  blue  or  green  due  to  the  different  magnification  and  focus 
between  the  three  bands. 

This  effect  is  not  noticeable  to  the  human  eye  at  normal  magnification,  but  is  very  obvious  when  the 
image  is  magnified.  It  is  also  obvious  when  looking  at  the  histogram.  The  histogram  of  a  black  and  white 
grid  would  be  expected  to  fall  on  a  straight  line  along  the  neutral  axis  of  the  color  cube,  as  illustrated  on  the 
left  of  figure  7.1.  Chromatic  aberration  causes  significant  deviations  from  the  neutral  line,  as  shown  in  the 
histogram  on  the  right. 

Chromatic  aberration  is  most  pronounced  where  there  are  sharp  edges.  And  that  causes  problem  not 
only  for  pictures  of  black  lines  on  a  white  background,  but  also  for  pictures  of  highlights.  For  smoother 
surfaces,  the  transition  between  highlight  and  non-highlight  areas  will  be  more  sudden,  and  thus  the  chromatic 
aberration  will  be  more  pronounced.  As  a  result,  the  highlight  cluster  of  histograms  for  smooth  objects  will 
be  more  widely  scattered  than  would  be  expected.  The  degree  of  this  scattering  depends  not  only  upon  the 
smoothness  of  the  object,  but  also  upon  the  characteristics  of  the  lens,  the  color  of  the  object,  and  where  in 
the  image  the  highlight  occurs.  (Chromatic  aberration  is  most  pronounced  in  the  periphery  of  the  image.) 
This  makes  it  very  difficult  to  model  in  a  general  way. 

This  thesis  takes  a  two-pronged  attack  upon  the  problem  of  chromatic  aberration.  The  first  step  is  to 
eliminate  as  much  of  the  chromatic  aberration  as  possible  at  the  time  the  image  is  taken.  This  is  done  by 
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active  lens  compensation.  Since  the  CE.  allows  fine  control  of  the  lens  position,  and  since  the  color  bands 
are  imaged  separately,  the  lens  is  refocused,  rezoomed,  and  repositioned  between  the  red,  green  and  blue 
color  bands.  This  approach  typically  reduces  the  chromatic  aberration  by  an  order  of  magnitude  (Novak  et 
al.,  1990],  [Willson  and  Shafer,  1991]. 

Unfortunately,  the  method  does  not  completely  eliminate  all  chromatic  aberration  in  images  with  high¬ 
lights.  Also  it  is  time-consuming  to  calibrate  and  is  not  generally  applicable  to  camera  setups  that  do  not  have 
a  finely  controlled  lens.  Therefore  the  second  approach  is  to  take  some  chromatic  aberration  into  account 
when  measuring  the  dimensions  of  the  color  histogram. 

Color  figure  A.8  shows  a  histogram  for  a  shiny  red  ball.  The  original  image  is  shown  in  the  upper  right 
hand  comer  of  the  image.  The  top  of  the  highlight  cluster  is  slightly  fanned  out,  making  it  slightly  wider 
than  the  bottom  of  the  highlight  cluster.  Color  figure  A.9  shows  the  histogram  associated  with  a  verj'  shiny 
red  pail.  Here  the  fan-out  is  even  more  pronounced.  (The  bottom  few  rows  of  the  original  image  where 
blooming  has  occurred  were  manually  eliminated  from  consideration.) 

This  fanning  of  the  highlight  cluster  is  particularly  troublesome  because  it  mns  completely  counter  to  the 
model  of  histogram  behavior.  The  model  says  that  th»e  will  be  more  variation  of  body  reflection  amounts 
and  the  base  of  the  highlight  cluster  than  at  the  upper  end.  A  highlight  cluster  that  is  wider  at  the  top  implies 
that  there  are  very  bright  highlight  pixels  occurring  at  several  different  reflection  angles,  but  that  for  some 
reason  pixels  with  lesser  amounts  of  highlight  occur  at  only  a  few  reflection  angles.  If  the  histogram  were 
to  be  believed,  then  the  amount  of  highlight  does  not  fall  off  as  the  off-specular  angle  is  increased.  This 
would  mean  a  rather  unusual  distribution  of  facet  angle.  A  more  plausible  explanation  of  the  fan-out  of 
the  highlight  cluster  is  that  the  chromatic  aberration  causes  large  hue  shifts  at  points  where  the  highlight  is 
increasing  very  rapidly. 

Since  the  chromatic  aberration  makes  the  highlight  cluster  wider  in  regions  where  the  highlight  increases 
rapidly,  this  would  imply  that  the  width  measurement  is  not  very  trustworthy  at  the  top  end  of  the  highlight 
cluster.  A  program  analyzing  the  histogram  in  figure  A.9  will  measure  a  very  wide  highlight  cluster,  even 
though  it  corresponds  to  a  very  shiny  object.  This  is  because  the  original  algorithm  examine  the  amount  of 
body  reflection  present  in  all  highlight  pixels  to  compute  the  width  measurement.  A  pixel  is  classified  as  a 
highlight  is  the  amount  of  surface  reflection  exceeds  the  threshold  4<Jc.  Although  the  model  predicts  that  the 
widest  part  of  the  highlight  cluster  is  at  the  base,  this  is  not  explicitly  tested  by  the  program. 
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Although  the  highlight  cluster  is  plagued  by  chromatic  aberration  at  the  top  end,  the  base  of  the  cluster 
is  still  well  behaved.  Now  that  chromatic  aberration  is  known  to  be  a  problem,  the  algonthm  measures 
width  over  all  highlight  pixels  near  the  threshold  point,  mstead  of  over  ail  pixels  that  exceed  the  threshold. 
Equation  5.8  that  gives  the  width  measurement  in  terms  of  the  computed  amounts  of  body  reflection  mt,  and 
surface  reflection  ntj  is  modified  to 

H’  =  [MAX{nib)  ~  MIN(nib)]ib  overall p  for  which  J  <  <  T  + 

Similarly,  the  second  width  measurement,  used  when  the  highlight  cluster  is  clipped,  becomes 
W2  =  [MAX(mb)  -  MIN(mh)]  /b  over  all  p  for  which  ST  <  m,  <  3T  + 

The  process  for  measuring  length  and  intersection  remain  unchanged. 


7.13  Blooming 

Color  figure  A.IO  shows  a  2x  magnification  of  the  lower  section  of  the  pail  image  in  figure  A,9.  This 
picture  clearly  shows  that  the  brightest  section  of  the  highlight  has  “bloomed”  into  neighboring  pixels.  The 
blooming  is  most  pronounced  in  the  red  band  of  the  image,  creating  a  red  halo  around  the  highlight.  This 
effect  is  not  chromatic  aberration  since  it  is  several  pixels  wide. 

Blooming  occurs  when  points  on  the  CCD  chip  are  so  saturated  by  incoming  photons  that  the  change 
in  one  well  on  the  chip  spills  over  into  neighboring  charge  wells.  Some  cameras  are  designed  to  be  fairly 
resistant  to  blooming,  an  advantage  when  taking  pictures  of  highhghts.  The  camera  used  to  take  the  pictures 
in  this  chapter  is  very  prone  to  blooming.  An  interesting  thing  to  note  is  that  it  blooms  predominantly  in  the 
horizontal  direction.  The  picture  in  figure  A.IO  is  only  a  mild  case  of  blooming.  In  some  cases  the  highlight 
will  smear  all  the  way  across  the  image. 

Bloomed  pixels  that  saturate  the  camera  do  not  present  a  problem  for  the  method  developed  in  this 
thesis.  Saturated  pixels  are  easy  to  detect;  any  pixel  with  a  value  equal  to  the  camera  maximum  is  considered 
tmtrustworthy  and  disregarded.  The  fact  that  the  object  is  too  bright  will  be  noted  and  the  clipped  method 
is  used  to  calculate  the  scene  parameters.  However  sometimes  blooming  will  increase  the  values  of  nearby 
pixels  without  causing  them  to  saturate.  This  means  the  pixel  values  may  be  inaeased  by  some  arbitrary 
amount,  to  some  new  arbitrary  value  that  is  less  than  the  camera  maximum.  Since  the  blooming  may  effect 
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very  large  areas  of  the  image,  it  is  difficuJt  to  handle  in  an  automatic  way.  In  the  histogram  for  figure  A.9, 
all  pixels  that  occurred  in  the  same  rows  as  bloomed  pixels  were  eliminated  from  consideration. 

Blooming  can  be  reduced  a  great  deal  by  manufacturing  a  camera  chip  with  charge  wells  further  apart. 
A  camera  of  this  type  would  make  is  possible  to  consider  all  regions  of  the  image  when  analyzing  the 
histogram,  instead  of  arbitrarily  throwing  away  portions  adjacent  to  saturated  pixels.  Saturated  pixels  do  not 
present  a  problem  for  the  method  developed  in  this  thesis,  provided  they  do  not  spill  charge  into  neighboring 
pixels. 

This  section  has  described  some  modifications  made  to  the  analysis  method  along  with  some  techniques 
for  taking  high  quality  images.  Taken  together,  they  allow  the  algorithm  to  work  on  real  images.  The  next 
few  sections  describe  the  performance  of  the  algorithm  in  estimating  phase  angle,  illumination  intensity,  and 
roughness  from  real  images. 


7.2  Estimating  Phase  Angle 


An  experiment  was  set  up  in  the  CIL  to  test  the  histogram  analysis  algorithm  at  estimating  phase  angle  from 
real  images.  A  series  of  images  was  taken  wdth  the  camera  and  light  source  separated  by  an  increasing  phase 
angle.  The  angle  was  measured  with  a  large  protractor  and  strings  to  indicate  the  direction  of  the  camera 
and  light  source.  A  diagram  of  the  setup  is  shown  in  figure  7.2.  The  angles  measured  by  this  method  were 
estimated  to  be  accurate  to  within  5  degrees.  The  light  was  a  250  Watt  spotlight;  it  was  estimated  to  appear 
a  few  degrees  across  at  the  distance  used  in  the  experiment  (about  5  feet).  Therefore  it  is  only  a  crude 
approximation  of  a  point  source. 

The  first  image  in  the  sequence  is  shown  in  color  figure  A.l  1.  This  picture  was  taken  when  the  camera 
and  light  source  were  approximately  10  degrees  apart.  The  phase  angle  was  then  increased  by  10  degrees 
between  each  picture.  The  last  image  in  the  sequence  is  shown  in  color  figure  A.  12.  This  picture  was  taken 
when  the  phase  angle  between  the  camera  and  light  source  was  90  degrees. 

The  program  automatically  spht  the  color  histograms  of  the  objects  into  two  clusters,  fit  lines  to  those 
vectors,  and  calculated  the  values  of  length,  width,  and  intersection.  The  color  histogram  of  the  image  in 
figure  A.12  is  shown  in  color  figure  A.13.  The  superimposed  white  lines  show  the  lines  fit  by  the  program. 
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Platform  for  holding  objects 


Protractor 


Light 
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Figure  7.2:  Experimental  setup  for  estimating  phase  angle 


5  0.9S 
O 

0.90 

c 

S  0.8S 

2  0.80 

£ 

5  0.75 

0.70 
0.65 
0.60 
0.55 
0.50 


+  + 


0  10  20  30  40  50  60  70  80  90 

phase  angle  between  light  and  camera 


Figure  7.3:  Plot  of  intersection  point  vs.  pha.se  angle 


This  process  was  repeated  for  each  image  in  the  sequence.  Figure  7.3  shows  the  measured  intersection 
value  for  each  image,  plotted  against  the  phase  angle  that  was  in  effect  at  the  time  the  image  was  taken.  A 
comparison  of  figure  7.3  with  figure  3.13  in  chapter  3  shows  that,  as  predicted,  the  intersection  decreases 
with  increasing  phase  angle.  The  sole  exception  is  at  6p  =  90°.  A  slight  error  in  setting  up  the  light  could 
explain  this  problem:  if  the  phase  angle  was  actually  slightly  larger  than  90°,  the  brightest  point  on  the 
objwt  would  be  out  of  view  of  the  camera,  throwing  off  the  intersection  ratio. 

The  same  polynomial  coefficients  used  in  section  6.2  were  used  to  calculate  the  phase  angle  from  the 
length,  width  and  intersection  measurements.  The  lookup  tables  in  that  section  assumed  a  camera  noise 
cTc  =  1.275  and  clipping  at  a  maximum  value  of  255.  The  results  are  shown  in  figure  7.4,  The  dotted  line 
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Figure  7.4;  Results  for  experiments  to  calculate  phase  angle  from  real  images 


shows  the  correct  answer,  using  the  phase  angle  measured  by  the  protractor  as  ground  truth.  The  average 
error  in  estimating  angle  is  9.96®. 

As  the  light  was  moved  around,  the  distance  from  the  object  did  not  remain  constant  due  to  the  layout  of 
the  lab.  Therefore  the  intensity  of  the  illumination  was  not  assiuned  to  be  constant.  For  two  of  the  images 
in  the  sequence,  the  illumination  was  close  enough  that  the  highlight  saturated  the  camera  and  clipping 
occurred.  This  happened  at  9p  =  60°  and  6p  =  80°.  In  these  two  cases  the  second  set  of  equations  {see 
equation  6.2)  for  clipping  were  used.  Although  this  technique  produced  a  reasonable  answer  for  the  case 
whim  dp  =  80°,  it  did  not  do  so  well  for  Op  -  60°.  If  this  data  point  is  eliminated  from  consideration,  the 
average  error  in  estimating  the  phase  angle  is  7.04°. 

Overall  the  method  developed  for  estimating  phase  angle  from  analyzing  color  histograms  works  fairly 
well,  especially  considmug  that  the  ground  truth  measurement  of  the  phase  angle  is  fairly  crude.  Also  the 
lookup  tables  in  section  6.2  were  calculated  without  calibrating  the  simulated  images  to  the  conditions  in 
the  CIL.  In  particular,  the  noise  of  the  camera  was  not  measured  precisely  and  the  light  source  used  in  the 
experiments  was  not  a  point  source  as  was  used  in  the  simulations.  Also,  the  Torrancc-Sparrow  roughness 
model  is  an  idealized  model  that  assumes  isotropic,  Gaussian  scattering;  it  may  not  describe  very  well  the 
roughness  of  the  real  object  used  in  these  experiments. 
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Figure  7.5;  Measured  illuminatiou  intensity  for  each  image  in  the  sequence 

7.3  Estimating  Illumination  Intensity 

A  second  experiment  was  performed  in  the  CDL  to  test  the  performance  of  the  algorithm  at  estimating 
illumination  intensity.  The  spotlight  was  plugged  into  a  variable  voltage  supply  with  a  manually  operated 
dial.  A  sequence  of  images  was  taken  under  increasing  levels  of  illumination,  while  the  imaging  geometry 
and  target  object  were  kept  constant.  Altogether  six  images  were  taken.  The  illumination  level  was 
measured  with  a  luminance  spot  meter  aimed  at  a  white  card.  A  graph  showing  the  luminance  measurement 
at  each  image  in  the  sequence  is  presented  in  figure  7.5.  The  units  are  candles  per  square  meter.  The  spot 
measurements  were  estimated  to  be  repeatable  to  within  5%. 

Again,  the  program  analyzed  the  histograms  to  produce  measurements  of  length,  width  and  intersection 
for  each  image.  The  polynomial  equation  to  calculate  illumination  intensity  was  then  applied  to  these 
measurements.  The  results  are  shown  in  figure  7.6.  The  horizontal  axis  shows  the  intensity  measured  by  the 
spot  meter,  while  the  vertical  axis  shows  the  intensity  estimated  by  the  histogram  analysis.  The  gain  of  the 
camera  has  not  been  calibrated,  so  the  program  gives  a  relative  estimate  of  intensity.  The  dotted  Ime  shows 
the  best  linear  fit  to  the  data.  If  the  slope  of  that  line  is  considered  to  be  the  gain  of  the  camera,  then  the 
average  error  in  estimating  illumination  intensity  is  5.07%. 

The  results  in  figure  7.6  were  computed  using  only  the  length,  width,  and  intersection  measurements  of 
the  highlight  cluster.  The  graph  in  figure  7.7  shows  the  roughness  estimate  for  each  image  in  the  sequence, 
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computed  at  the  same  time  as  the  intensity  estimat'^s.  Figure  7.8  shows  the  estimated  phase  angle  for  the 
same  sequence.  Both  measurements  are  fairly  stable  aaoss  the  sequence  as  would  be  expected  since  the 
imaging  geometry  and  target  object  were  the  same  in  every  case.  The  graph  in  figure  7.7  is  particularly 
important,  since  both  increasing  smoothness  and  increasing  illumination  intensity  will  lengthen  the  highlight 
cluster.  These  graphs  show  that  for  the  most  part,  the  algorithm  does  not  confuse  changes  in  the  histogram 
shape  due  to  increasing  intensity  levels  with  the  effects  of  roughness  and  phase  angle. 

The  ground  truth  measmements  for  this  experiment  came  firom  a  luminance  meter,  which  were  compared 
with  estimates  of  intensity.  Unfortunately  these  measurements  are  somewhat  different.  Luminance  is 
measured  by  integrating  with  the  human  luminous  efficiency  function  V(A),  shown  in  figure  7.9.  A  luminance 
meter  shows  how  bright  a  spot  will  appear  to  humans,  but  not  necessarily  how  bright  it  will  appear  to  a  color 
camera.  A  narrow  band  red  illumination  will  have  a  much  lower  luminance  than  a  narrow  band  green  one. 
yet  both  may  have  the  same  intensity  as  measured  by  a  color  camera. 

The  difference  between  luminance  and  intensity  is  relevant  to  the  results  presented  here,  because  as  the 
voltage  fed  to  the  spotlight  is  decreased,  the  light  becomes  noticeably  redder  in  hue  along  with  becoming 
dimmer.  This  means  that  luminance  may  not  correlate  well  with  the  intensity  in  this  experiment.  While  the 
histogram  analysis  did  correctly  calculate  increasing  illumination  intensities  as  the  light  level  was  increased. 


122 


CHAPTER  7.  APPUCATION  OF  THE  ALGORITHM  TO  REAL  IMAGES 


Figure  7.7:  Calculating  roughness  when  intensity  changes 


Figure  7.8:  Calculating  phase  angle  when  intensity  changes 
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the  calculated  values  do  not  appear  as  linear  as  one  might  hope.  The  use  of  a  luminance  meter  might  be 
causing  some  of  the  problems. 


7.4  Estimating  Roughness 


A  third  experiment  was  performed  to  show  how  the  system  estimates  surface  roughness  from  color  nis- 
tograms.  Color  figure  A. 14  shows  an  image  of  a  plastic  inflated  pool  toy  in  the  shape  of  an  alligator.  Color 
figure  A.  15  is  an  image  of  a  plastic  pumpkin  for  trick-or-treating  The  pictures  of  the  terra-cotta  ball  in 
figure  A.1 1 ,  the  red  plastic  ball  in  figure  A.8,  and  and  red  plastic  pail  in  figure  A.9  were  also  analyzed  for 
this  experiment. 

Figure  7.10  shows  the  roughness  calculated  by  the  system  for  each  of  these  objects.  The  objects  are  listed 
in  order  of  decreasing  roughness,  as  estimated  by  human  observation.  The  calculated  roughness  number  is 
the  standard  deviation  of  facet  emgles,  in  degrees. 

There  is  no  error  measure  for  these  results,  since  there  is  no  ground  truth  data  for  the  actual  roughness 
values.  Nevertheless,  the  roughness  ranking  from  the  program  agrees  with  that  produced  by  a  human 
observer.  The  pumpkin  presents  a  particularly  interesting  case,  since  it  shows  roughness  at  more  than  one 
scale.  At  the  large  scale  where  roughness  is  judged  by  touch,  it  is  clearly  the  roughest  object.  This  roughness 
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Object 

Calculated  Roughness 

Alligator 

10.07= 

Pumpkin 

8.93= 

Terra-cotta  ball 

3.61  = 

Red  ball 

0.40= 

Red  pail 

0.10= 

Figure  7.10:  Table  of  estimating  roughness  results 

is  large  enough  to  be  considered  “texture”.  At  the  smaller  scale  of  optical  roughness,  it  is  considered  to  be 
more  shiny  than  the  alligator.  All  the  images  are  shown  together  in  color  figure  A.  16  for  better  comparison. 

The  result  of  this  chapter  is  that  the  method  developed  in  this  thesis  can  be  applied  to  real  images. 
The  algorithm  is  produces  reasonable  estimate  of  phase  angle,  illumination  intensity,  and  surface  roughness 
by  analyzing  the  dimensions  of  color  histograms.  The  method  is  able  to  succeed  even  with  such  camera 
problems  as  noise,  clipping,  and  chromatic  aberration.  Furthermore  the  method  is  independent  of  the  shape 
of  the  object,  and  works  on  shapes  ranging  from  a  pumpkin  to  an  alligator.  The  model  used  to  develop 
the  lookup  tables  is  fairly  general,  and  was  not  calibrated  to  match  the  actual  imaging  conditions  such  as 
light  source  extent,  camera  noise  characteristics,  actual  distribution  of  facet  angles,  etc.  It  is  likely  that  the 
method  would  perform  even  better  if  the  lookup  tables  used  measurements  from  real  histograms  rather  than 
from  simulated  ones.  Nevertheless,  the  method  works  reasorutbly  w'ell,  even  without  prior  reference  to  real 
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Summary  and  Conclusions 


This  thesis  describes  the  relationship  between  scene  parameters  and  the  color  variation  on  an  object.  This 
variation  is  analyzed  in  a  systematic  way  by  examining  the  shape  of  the  color  histogram  formed  by  the 
variation.  Equations  that  describe  the  physical  laws  of  reflection  lead  naturally  to  a  mathematical  relationship 
between  scene  properties  and  histogram  shape.  By  characterizing  the  shapie  of  histogram  clusters  m  terms 
of  simple  measurements  such  as  length,  width,  direction,  and  point  of  intersection,  this  thesis  shows  that  it 
is  possible  to  recover  estimates  of  scene  parameters  from  a  single  color  histogram.  The  resulting  algorithm 
works  very  quickly  and  requires  only  a  single  color  image. 


8.1  Contributions 

This  thesis  make  contributions  to  several  aspects  of  computer  vision,  including  highlight  analysis,  roughness 
estimation,  and  color  constancy. 

•  This  thesis  shows  that  color  variation  has  a  direct,  quantitative  relationship  to  scene  properties. 

Traditional  methods  treat  color  variation  as  a  statistical  quantity,  or  as  a  source  of  noise  to  be  overcome. 
This  thesis  shows  that  the  color  variation  may  be  modeled  as  a  regular  process  that  has  structure  which 
can  be  exploited  to  recover  information  about  the  scene. 

•  The  physical  laws  of  reflection  determine  that  relationship. 
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Physical  laws  show  that  inhomogeneous  dielectrics  have  two  types  of  reflection:  body  reflection 
and  surface  reflection.  In  most  cases,  the  two  different  processes  of  reflection  have  differ¬ 
ent  spectral  distributions,  so  that  the  color  vanaiion  will  have  a  planar  distribution  in  color 
space.  However  the  two  different  processes  also  have  different  geometrical  distributions.  These 
different  distributions  give  rise  to  a  particular  shape  of  histogram  within  the  plane. 

•  The  color  variation  is  related  to  both  color  and  non-color  properties. 

It  is  obvious  that  the  colors  recorded  by  the  camera  are  related  to  the  color  of  the  objects  m  the 
scene.  The  Dichromatic  Reflection  Model  also  shows  how  these  colors  are  related  to  the  color  of 
the  illumination.  However,  this  thesis  shows  how  the  color  variation  is  rehied  to  the  non-color 
properties  oi  surface  roughness  and  the  phase  angle  between  the  camera  and  light  source. 

•  This  thesis  presents  a  method  for  estimating  roughness  that  is  independent  of  the  object  shape. 

The  physical  laws  of  reflection  show  that  colors  seen  on  rough  surfaces  are  not  present  on  smooth 
surfaces  and  vice  versa.  This  means  that  roughness  is  an  important  factor  in  explaining  color  appear¬ 
ance.  It  also  means  that  color  variation  may  by  analyzed  to  recover  estimates  of  surface  roughness. 
The  results  in  this  thesis  show  that  the  method  works  for  sphere-shaped  objects,  but  also  for  surfaces 
with  irregular  shapes,  such  as  a  pumpkin  or  alligator. 

•  This  thesis  shows  how  imaging  geometry  affects  color  appearance. 

Colors  seen  under  one  imaging  geometry  may  not  be  present  for  different  imaging  geometries.  As 
the  camera  and  light  source  are  separated  by  increasing  phase  angles,  the  amounts  of  body  reflection 
and  surface  reflection  at  each  point  on  the  surface  will  shift,  but  in  different  ways.  Some  of  the  colors 
observed  when  the  camera  and  light  source  are  close  together  (small  phase  angle)  will  not  be  present 
when  the  camera  and  light  source  are  far  apan.  This  contributes  to  the  difflculties  caused  by  highlights 
in  such  tasks  as  stereo  matching.  When  the  viewing  angle  changes,  not  only  does  the  location  of  the 
highlight  on  the  object  change,  but  also  the  actual  colors  seen  in  it. 

•  There  is  a  difTerence  in  color  appearance  between  a  rough  object  illuminated  by  a  bright  source, 
and  a  smooth  object  illuminated  by  a  dim  source. 

Previous  methods  to  analyze  color  were  only  able  to  recover  relative  measurements  of  illumination 
color.  However,  by  considering  the  dichromatic  components  of  reflection,  this  thesis  shows  that  the 
shape  of  the  color  histograms  may  be  analyzed  to  recover  estimates  of  both  roughness  and  intensity. 
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Experiments  demonstrate  that  the  method  is  able  to  detect  changes  in  illumination  intensity,  and  is 
able  to  distinguish  them  from  changes  in  surface  roughness. 

•  Interreflection  also  has  a  regular,  quantifiable  effect  upon  the  color  histogram. 

Interreflection  occurs  even  in  simple  scenes  with  a  small  number  of  objects.  It  can  have  a  pronounce'^ 
effect  upon  the  image  appearance,  and  also  upon  the  structure  of  the  histogram.  This  thesis  shows 
how  the  Dichromatic  Reflection  Model  can  be  extended  to  cover  interreflection.  Considering  each 
component  of  reflection  and  interreflection  separately  makes  it  possible  to  quantify'  the  effects  in  a 
straightforward  manner. 

8.2  Directions  for  Future  Research 

The  results  presented  in  this  thesis  demonstrate  the  effectiveness  of  analyzing  color  histograms  to  estimate 
scene  properties.  There  are  several  ways  in  which  the  method  could  be  extended. 

The  model  used  in  this  thesis  mrkes  several  limiting  assumptions,  and  although  they  do  not  prevent  the 
method  from  producing  reasonable  results,  it  is  likely  that  the  results  could  be  even  better  if  more  realistic 
assumptions  are  incorporated  into  the  model. 

For  example,  the  illunoination  is  modeled  as  a  single  point  light  source.  There  are  very  few  real  images 
for  which  this  model  captures  the  nature  of  the  illumination.  Some  ambient  illumination  is  present  in  most 
real  images,  since  there  is  usually  interreflection  from  the  floor,  ceiling,  and  walls.  Also,  typical  light  sources 
have  some  extent,  since  most  point  light  sources  do  not  emit  enough  light  to  take  good  pictures.  The  light 
used  to  take  the  pictures  in  chapter  7  was  not  a  point  light  source,  and  it  is  possible  that  the  estimates  of 
scene  parameters  could  be  improved  if  this  were  taken  into  accoimt  in  the  model. 

Ambient  light  in  the  scene  will  have  the  effect  of  shifting  the  whole  histogram  away  from  the  black  point 
of  the  color  cube.  The  body  reflection  cluster  would  begin,  not  at  the  black  point,  but  at  some  brighter  point 
that  depends  upon  the  ambient  illumination.  This  might  require  very  little  adjustment  to  the  algorithm:  the 
length  of  the  body  reflection  cluster  could  be  measured  from  the  maximum  down  to  the  ambient  point  rather 
than  down  to  the  black  point  of  the  color  cube. 

An  extended  light  source  will  have  the  eff-cf  of  making  the  highlight  cluster  wider  than  it  would  have 
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been  for  a  point  source.  This  is  because  the  number  of  reflectance  angles  on  the  object  where  the  camera 
and  light  source  are  lined  up  to  produce  a  highlight  will  be  greater.  The  extent  of  a  light  source  can  be 
incorporated  into  the  simulation  that  creates  the  lookup  tables.  However,  this  assumes  that  the  same  size 
light  source  will  be  used  every  time,  so  this  approach  is  not  completely  general.  A  more  general  method 
might  be  to  estimate  the  extent  of  the  light  source  by  measuring  the  width  of  the  highlight  cluster  at  the  upper 
end,  where  the  highlight  is  at  a  maximum,  and  incorporate  this  estimate  into  the  algorithm. 

A  further  extension  to  the  model  would  be  to  handle  multiple  extended  sources  of  the  same  color.  This 
type  of  lighting  is  common  in  many  situations,  such  as  office  and  home  interiors.  If  the  light  sources  are  at 
different  phase  angles  with  respect  to  the  camera,  the  resulting  highlights  will  occur  at  different  reflectance 
angles  on  the  object.  Just  as  the  interreflection  can  produce  a  second  highlight  cluster,  each  of  the  extended 
light  sources  would  produce  a  separate  highlight  cluster  whose  intersection  with  the  body  reflectance  cluster 
depends  upon  the  phase  angle  for  that  light. 

A  final  extension  to  the  illumination  model  is  the  analysis  of  histograms  where  interreflection  is  present. 
Interreflection  presents  the  most  genera!  model  for  illumination,  since  it  can  act  like  ambient  illumination 
(interreflection  from  walls);  or  multiple  extended  light  sources  (surface  reflection  from  a  nearby  object);  but 
can  also  affect  the  appearance  in  a  completely  different  maimer  (e.g.  body-to-body  reflection).  Chapter  4 
showed  how  interreflection  from  a  nearby  object  affects  the  appearance  of  the  histogram,  but  did  not  present 
a  method  for  automatically  analyzing  images  with  interreflection.  Thus  a  direction  of  future  work  is  to 
turn  the  predictive  model  into  an  algorithm,  as  has  already  been  done  for  the  simple  histogram  without 
interreflection. 

The  structure  of  the  histogram  that  is  described  in  this  thesis  assumes  that  objects  are  small  in  comparison 
to  the  distance  to  the  camera  and  light  source,  which  is  the  orthographic  model.  Under  this  assumption,  the 
phase  angle  between  the  camera  and  light  source  is  the  same  for  all  points  on  the  object,  so  that  highlights 
always  occur  at  the  same  reflectance  angle,  even  for  objects  with  multiple  highlights.  While  the  orthographic 
assumption  is  a  good  approximation  for  some  images,  it  is  a  poor  model  for  images  where  objects  are  large 
compared  with  their  distance  from  th  j  camera.  In  such  situations  a  perspective  model  should  be  used. 
Therefore  another  direction  for  future  research  is  to  examine  how  the  shape  of  the  histogram  is  affected  by 
perspective  projection. 

Another  simplifying  assumption  made  in  this  work  is  about  the  nature  of  the  surface  roughness.  The 
Torrance-Spairow  model  assumes  isotropic,  Gaussian  distribution  of  facet  slopes.  This  type  of  scattering 
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is  likely  to  characterize  the  differences  between  gloss,  semi-gloss  and  eggshell  paint  finishes,  where  the 
roughness  is  caused  by  a  deliberate  process.  However  it  may  not  describe  very  well  roughness  caused  by 
wear  or  age.  It  is  likely  that  scattering  caused  by  these  latter  processes  will  have  a  preferred  direction.  A 
direction  for  future  research  is  the  extension  of  the  current  model  to  handle  non-isotropic  scattering  and 
non-Gaussian  distributions  of  facet  angles. 

The  method  presented  here  produces  reasonable  results  on  real  images,  even  though  the  measurements 
for  the  lookup  table  came  from  simulated  data.  However,  it  is  possible  that  better  results  could  be  achieved 
by  using  histogram  measurements  from  real  images  to  fill  the  lookup  tables.  This  type  of  data  would  require 
calibrated  samples  of  roughness  which  are  difficult  to  obtain  However,  it  would  provide  the  benefit  of 
ground-truth  data  on  the  effect  of  surface  roughness  on  histogram  appearance.  Measurements  from  real, 
calibrated  images  would  also  mean  that  the  lookup  tables  will  accurately  reflect  the  changes  in  histogram 
shape  due  to  changes  in  illumination  intensity  and  imaging  geometry.  Finally,  histograms  from  real  images 
do  not  have  the  weakness  of  relying  on  idealized  models  of  reflection,  such  as  Lambertian  shading.  Therefore 
another  direction  of  future  research  is  comparison  of  the  models  used  here  with  actual  images  of  known 
parameters. 


8.3  Applications 

One  possible  application  for  the  method  described  in  this  thesis  is  automated  inspection.  The  roughness  of 
surfaces  may  be  an  important  measurement  for  determining  quality  in  manufacturing  new  objects,  where 
flaws  in  the  finish  may  cause  scattering  of  surface  reflection  [Cielo,  1988].  Roughness  estimation  might  also 
be  useful  for  detecting  excessive  wear  in  frequently  used  parts,  where  abrasions  cause  a  formerly  smooth 
object  to  become  rough  in  places.  The  fact  that  the  method  presented  here  works  quickly,  and  from  a 
single  color  image,  indicates  that  it  may  prove  useful  for  moving  assembly  lines  or  other  setups  where  the 
inspection  must  be  done  in  a  short  period  of  time. 

Another  application  of  the  type  of  analysis  performed  by  this  thesis  is  in  computing  color  constancy.  The 
colors  in  a  scene  measured  by  a  camera  change  whenever  the  illumination  color  changes,  so  it  is  useful  to 
be  able  to  compute  the  illumination  color  and  the  colors  of  objects  in  the  scene  as  they  would  appear  under 
“while”  light.  Previous  methods  for  computing  illumination  color  from  highlights  rely  on  the  intersection 
of  more  than  one  dichromatic  reflection  plane  to  determine  the  illumination  color  [Klinker  et  al.,  1988], 
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[Lee,  1986],  or  cannot  determine  illumination  intensity  (Lee,  1988].  By  modeling  the  effects  of  the  body 
reflection  and  surface  reflection  distributions,  the  method  presented  in  this  thesis  is  able  to  estimate  both 
illumination  chromaiiciry  and  intensity,  even  when  only  a  single  surface  with  highlights  is  present.  Color 
constancy  is  useful  for  tasks  ranging  from  target  recognition  for  moving  robots,  to  automated  film  processing 
and  printing. 

Estimates  of  the  phase  angle  between  the  camera  and  the  light  source  seem  less  intnnsically  useful  at 
first.  However  there  is  an  application  to  automated  film  processing;  automatically  determining  whether  or 
not  a  flash  was  used  in  taking  the  picture.  A  built-in  flash  means  that  the  phase  angle  between  the  camera 
direction  and  the  illumination  direction  will  be  small.  A  large  phase  angle  means  that  it  is  very  unlikely 
that  a  flash  was  used.  (An  exception  to  this  rule  is  provided  by  professional  photographers  who  use  remote 
flashes,  but  they  are  unlikely  to  trust  their  film  to  automated  processing  in  any  case.) 

Estimates  of  the  phase  angle  may  also  be  useful  for  estimating  the  amount  that  highlights  are  likely  to 
shift  between  images  in  a  stereo  pair.  The  shift  in  the  highlight’s  position  on  the  object  is  likely  to  be  small 
if  the  phase  angles  for  the  cameras  are  similar.  Also,  the  phase  angle  determines  how  much  the  highlight 
colors  will  change  between  stereo  pairs.  Changes  in  the  intersection  of  the  highlight  cluster  are  equivalent 
to  changes  in  the  colors  observed  in  the  highlight.  The  mtersection  of  the  highlight  cluster  changes  slowly 
for  small  phase  angles,  and  more  quickly  for  large  phase  angles.  Estimates  of  phase  angle  could  be  used  to 
give  stereo  matching  algorithms  error  tolerances  in  matching  positions  and  colors  of  highlights. 


8.4  Concluding  Remarks 


The  insights  presented  by  this  thesis  have  application  to  the  general  field  of  image  understanding.  This  work 
has  shown  that  not  only  do  the  colors  of  object  and  illumination  affect  color  appearance  of  shiny  objects, 
but  also  that  the  non-color  parameters  of  surface  roughness  and  imaging  geometry  play  an  important  role 
in  color  appearance.  The  relationship  between  scene  parameters  and  color  variation  is  made  explicit  by 
examining  physical  models  of  reflection  for  each  of  the  dichromatic  components.  A  surprising  amount  of 
information  can  be  recovered  from  examining  a  single  color  histogram.  This  indicates  how  complex  and  yet 
how  rich  the  process  of  vision  is,  and  why  color  is  particularly  valuable  for  image  understanding. 


Appendix  A 


Color  Figures 
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Figure  A.3;  Simulation  results 
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Figure  A.4:  Simulated  test  image 


Figure  A.5:  Image  simulated  from  recovered  parameters 


Figure  A.6:  Intrinsic  images  calculated  with  a  skewed  highlight  color 


Figure  A.7:  Intrinsic  images  calculated  with  the  corrected  highlight  color 
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Figure  A.IO:  Closeup  of  bloomed  area 


Figure  A.l  1:  Image  with  phase  angle  of  10  degrees 


Figure  A,  12;  Image  with  phase  angle  of  90  degrees 


Figure  A.  13:  Histogram  for  phase  angle  of  90  degrees 
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Figure  A.  14:  Image  of  plastic  alligator 


Figure  A,  IS:  Image  of  plastic  pumpkin 


Figure  A.  16:  Five  objects  with  different  roughness  values 
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